Contactless cough detection and attribution

ABSTRACT

Methods, devices, and systems for contactless cough detection and attribution are presented herein. Audio data may be received using a microphone. A cough may be identified as having occurred based on the received audio data. Radar data may be received indicative of reflected radio waves from a radar sensor. A state analysis process may be performed using the received radar data. The detected cough may be attributed to a particular user based at least in part on the state analysis process performed using the radar data.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/990,720, filed Aug. 11, 2020, entitled “Contactless Cough Detectionand Attribution.” This application is related to PCT ApplicationUS2019/031,290, filed May 8, 2019, entitled “Sleep Tracking and VitalSign Monitoring Using Low Power Radio Waves.” This application is alsorelated to Non-Provisional application Ser. No. 16/990,705, filed Aug.11, 2020, entitled “Contactless Sleep Detection and DisturbanceAttribution.” This application is also related to Non-Provisionalapplication Ser. No. 16/990,714, filed Aug. 11, 2020, entitled“Contactless Sleep Detection and Disturbance Attribution for MultipleUsers.” This application is also related to Non-Provisional applicationSer. No. 16/990,746, filed Aug. 11, 2020, entitled “Initializing SleepTracking on a Contactless Health Tracking Device.” This application isalso related to Non-Provisional application Ser. No. 16/990,726, filedAug. 11, 2020, entitled “Precision Sleep Tracking Using a ContactlessSleep Tracking Device.” The entire disclosures of which are herebyincorporated by reference for all purposes.

BACKGROUND

A person may have little insight into his or her behavior at night. Forexample, a person may cough some number of times during the night. Theperson may be unaware of whether he is coughing. In order to improve theperson's sleep quality, by the person learning what is disturbing hissleep, the person may be able to take preventative measures to addressthe source and decrease the presence or effect of future disturbances.

SUMMARY

Various embodiments are described related to a contactless coughdetection device. In some embodiments, a contactless cough detectiondevice is described. The device may comprise a housing. The device maycomprise a microphone housed by the housing. The device may comprise aradar sensor housed by the housing. The device may comprise a processingsystem housed by the housing, comprising one or more processors that mayreceive data from the microphone and the radar sensor. The processingsystem may be configured to receive audio data from the microphone. Theprocessing system may be configured to detect that a cough has occurredbased on the received audio data. The processing system may beconfigured to receive radar data indicative of reflected radio wavesfrom the radar sensor. The processing system may be configured toperform a sleep state analysis process using the received radar data.The processing system may be configured to attribute the detected coughto a particular user based at least in part on the sleep state analysisprocess performed using the received radar data.

Embodiments of such a device may include one or more of the followingfeatures: the processing system may detect that the cough has occurredby analyzing the received audio data from the microphone using apre-trained cough detection machine learning model. The processingsystem may be further configured to delete the audio data received fromthe microphone after detecting that the cough has occurred. Theprocessing system being configured to attribute the detected cough tothe particular user may comprise the processing system being configuredto determine that the only user being monitored may be attributed withthe detected cough. The processing system being configured to performthe sleep state analysis process may comprise the processing systembeing configured to determine the particular user has moved within a bedwithin a time period of the detected cough. The processing systemattributing the detected cough to the particular user may comprise theprocessing system determining that the particular user of a plurality ofusers being monitored may be attributed with the detected cough.

The processing system being configured to perform the sleep stateanalysis process may comprise the processing system being configured todetermine the particular user has moved more than other users of theplurality of users within a bed within a time period of the detectedcough. The processing system may be further configured to cause sleepdata to be stored indicative of the cough being attributed to theparticular user. The device may further comprise a wireless networkinterface housed by the housing and in communication with the processingsystem. The device may further comprise a speaker housed by the housingand in communication with the processing system. The processing systemmay be further configured to receive a spoken command, via themicrophone. The processing system may be further configured to outputdata based on the spoken command, via the wireless network interface, toa cloud-based server system. The processing system may be furtherconfigured to, in response to the output data, receive an instructionfrom the cloud-based server system via the wireless network interface.The processing system may be further configured to output stored sleepdata in response to the instruction. The processing system may befurther configured to output a sleep report that indicates a number oftimes that the particular user coughed during sleep. The device mayfurther comprise an electronic display, in communication with theprocessing system, that may output the sleep report for presentation.The processing system may be further configured to create, over aplurality of days, a trend report indicating whether an amount ofcoughing by the particular user may be increasing, decreasing, orstaying constant. The radar sensor may be an integrated circuit (IC)distinct from the processing system. The IC outputs frequency-modulatedcontinuous wave (FMCW) radio waves into an environment of thecontactless cough detection device. The FMCW radar may have a frequencybetween 57 and 64 GHz, may have a peak effective isotropically radiatedpower (EIRP) of 20 dBm or less, and may have a targeting regionsufficient to encompass an area of a multiple-user bed.

In some embodiments, a method for performing contactless cough detectionis described. The method may comprise receiving an audio data stream.The method may comprise detecting that a cough may have occurred basedon the received audio data stream. The method may comprise receiving aradar data stream. The method may comprise performing a sleep stateanalysis process using the received radar data. The method may compriseattributing the detected cough to a particular user based on the sleepstate analysis process performed using the received radar data.

Embodiments of such a method may include one or more of the followingfeatures: detecting that the cough has occurred may be performed byanalyzing the received audio data stream using a pre-trained coughdetection machine learning model. The method may further comprisedeleting the received audio data stream after detecting that the coughhas occurred. Performing the sleep state analysis process may comprisedetermining that the particular user has moved within a bed within atime period of the detected cough. Performing the sleep state analysisprocess may comprise determining that the particular user has moved morethan one or more other users within a bed within a time period of thedetected cough. Attributing the detected cough to the particular usermay comprise determining that the particular user may be attributed withthe detected cough. A processing system of a contactless cough detectiondevice may receive the audio data stream from a microphone and the radardata stream from a radar integrated circuit (IC) of the contactlesscough detection device. The method may further comprise receiving aspoken command via a microphone. The method may further compriseoutputting data based on the spoken command, via a wireless networkinterface, to a cloud-based server system. The method may furthercomprise receiving an instruction from the cloud-based server system viathe wireless network interface in response to the output data. Themethod may further comprise outputting stored sleep data in response tothe received instruction via an electronic display.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of variousembodiments may be realized by reference to the following figures. Inthe appended figures, similar components or features may have the samereference label. Further, various components of the same type may bedistinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

FIG. 1 illustrates an embodiment of a system for performing contactlesssleep tracking.

FIG. 2A illustrates an embodiment of a sleep tracking system.

FIG. 2B illustrates an embodiment of a sleep tracking system withintegrated beam targeting.

FIG. 2C illustrates an embodiment of frequency-modulated continuous waveradar radio waves output by a radar subsystem.

FIG. 3A illustrates an embodiment of a contactless sleep trackingdevice.

FIG. 3B illustrates an exploded view of an embodiment of a contactlesssleep tracking device.

FIG. 4 illustrates a cross-sectional view of a contactless sleeptracking device.

FIG. 5 illustrates an embodiment of a state machine for determining whena person is sleeping.

FIG. 6 illustrates timelines of detected sleep states and environmentaldisturbances.

FIG. 7 illustrates an embodiment of waveform data in which movement dueto vital signs of a user are observable.

FIG. 8 illustrates an embodiment of a method for performing contactlesssleep detection and disturbance attribution.

FIG. 9 illustrates an embodiment of a contactless sleep tracking devicemonitoring multiple users.

FIG. 10 illustrates an embodiment of a sleep tracking system that cantrack multiple users.

FIGS. 11A and 11B illustrate graphs of detected movement at variousdistances.

FIG. 12 illustrates a graph of detected movement being split intomultiple targets.

FIG. 13 illustrates an embodiment of a method for performing sleeptracking for multiple users.

FIG. 14 illustrates an embodiment of a beam-steering module for acontactless sleep tracking device that targets the direction in whichsleep tracking is performed.

FIG. 15 illustrates an embodiment of the antenna layout of a radarsubsystem that may be used in combination with a beam-steering module ofa contactless sleep tracking device.

FIG. 16 illustrates another embodiment of a beam-steering module fortargeting the direction in which sleep tracking is performed.

FIG. 17 illustrates an embodiment of a method for directionallytargeting sleep tracking.

FIG. 18 illustrates a cough detection and attribution device.

FIG. 19 illustrates examples of timelines of detected coughs and sleepdisturbances for a single monitored user.

FIG. 20 illustrates examples of timelines of detected coughs and sleepdisturbances for multiple monitored users.

FIG. 21 illustrates an embodiment of a method for cough detection andattribution.

FIG. 22 illustrates an embodiment of a sleep tracking system thatperforms a sleep setup process.

FIG. 23 illustrates an embodiment of a first instructional userinterface presented during a sleep setup process.

FIG. 24 illustrates an embodiment of a second instructional userinterface presented during a sleep setup process.

FIG. 25 illustrates an embodiment of a third instructional userinterface presented during a sleep setup process.

FIG. 26 illustrates an embodiment of a user interface presented during asleep setup process.

FIG. 27 illustrates an embodiment of a user interface presentedfollowing a successful sleep setup process.

FIG. 28 illustrates an embodiment of a user interface presentedfollowing an unsuccessful sleep setup process.

FIG. 29 illustrates another embodiment of a user interface presentedfollowing an unsuccessful sleep setup process.

FIG. 30 illustrates an embodiment of a method for performing an initialsetup process of a sleep tracking device.

DETAILED DESCRIPTION

Embodiments detailed herein are focused on performing contactless sleepmonitoring, attributing causes of sleep interruptions, and systems anddevices to perform contactless sleep monitoring and analysis. A singledevice may be situated bedside. For some embodiments, it is preferredthat the single device makes no physical contact with a user or theuser's bed. The device, without any physical contact, may monitor theuser to assess whether the user is awake or asleep while in bed. Whenthe user transitions from being asleep to awake, the device maydetermine what caused the user to wake. In addition to performing sleepmonitoring, the device may monitor one or more environmental conditions,such as ambient sound, light, and temperature. If a loud enough sound,increase in lighting, and/or significant enough change in temperatureare detected around the time that the user awoke, the environmentalcondition may be identified as the cause of the user waking.

Throughout one or more nights, the contactless sleep analysis device maymonitor the user to determine when the user awoke during the night andwhat environmental condition may be to blame. When the user is awake,the user may be provided information indicative of when they awoke, howoften they awoke and/or what, if any, environmental conditions were thelikely causes of the user waking. If a same environmental condition isrepeatedly attributed with causing the user to wake, the device mayprovide a recommendation that the user try to eliminate or decrease thepresence of the environmental condition. For instance, if the lightinglevel in the user's sleeping environment tends to increase just beforethe user awakes, the user should address the cause of the lighting toimprove their sleep. For instance, car headlights shining in a window ora display screen activating may be causing the light. To remedy, theuser could adjust their window dressings or power down their displayscreen, respectively.

Detection of whether the user is asleep or awake may be accomplishedusing low-powered radar. Low-powered radar, which can involve the use offrequency-modulated continuous wave (FMCW) radar, may involve thecontactless sleep analysis device emitting CW radar towards the user'sbed. Reflected radio waves may be analyzed to determine a distance tothe object that caused the reflection and a phase shift in the reflectedradio waves. Large movements detected using radar may be used todetermine if the user is awake or sleeping. Small movements may be usedto measure the user's vital signs, such as heartrate and breathing rate.

FMCW can be particularly effective in observing vital signs of a user.Generally, FMCW allows for finer measurement of movement compared withultra wideband (UWB). For example, devices based on UWB may be able todetect movement of 10 mm at a distance of 3 m, but FMCW devices may beable to measure movement of 2 mm at a similar distance. To realize thisadvantage, FMCW allows for the phase shift of emitted radio waves to bemeasured to detect small displacements of an object.

The contactless sleep analysis device may be highly privacy-preserving.Sleep data might not be collected without explicit permission beinggranted by a user. When sleep data is being collected, an indication assuch may be presented on the display. In some embodiments, sleep-relateddata is not transmitted to a remote server. Rather, in such embodiments,the sleep data is available only at the device locally. In someembodiments, the user may be required to give explicit consent for anysleep-related data to be transmitted to a remote server for storageand/or analysis. In some embodiments, no identity of the user is storedwith the sleep data; therefore it might not be possible to ascertain towhom the sleep data corresponds without additional information.

In some embodiments, sleep analysis may be performed on more than oneuser concurrently. In such multiple user arrangements, two users may bethe most likely arrangement (e.g., two spouses); however, three or moreusers may also be possible (e.g., two spouses and small child). Such anarrangement can allow for a sleep analysis to be performed individuallyfor each user. The contactless sleep analysis device may store or outputsleep data for each user separately and may provide each user with aseparate sleep report, which may be indicative of environmental eventsthat likely caused the individual user to awake. Therefore,advantageously, despite a single contactless sleep analysis device beingpresent, multiple users sleeping in a same bed may have their sleepseparately monitored over a same time period.

As previously detailed for a single user, environmental factors may bemonitored using one or more environmental sensors. Environmental factorsmay be monitored to determine if an environmental factor caused a userto wake. Additionally, a user waking may be attributed to another usermoving. For instance, if a first user rolls over in bed, this motion maybe identified as the environmental factor that caused the second user towake.

Depending on the setup of a user's bedroom, the direction from thecontactless sleep analysis device to where the one or more users sleepmay vary. For example, a user may have the contactless sleep analysisdevice on a nightstand that is taller than the user's bed, while anotheruser may have the sleep device on a nightstand that is the same heightor shorter than the user's bed. Additionally or alternatively, thecontactless sleep analysis device may be rotated horizontally at anangle to where the user sleeps. In some or all of the detailedembodiments herein, the direction monitored by the contactless sleepanalysis device may be targeted vertically and/or horizontally towardthe one or more users.

A beam-steering module, which may perform preprocessing on receiveddigital data from a radar subsystem, may perform weighted delay and sum(WDAS) beam-steering. Depending on the number and location of antennasthrough which reflected radio waves are sensed by the radar subsystem,targeting may be performed vertically and/or horizontally. Beam-steeringmay take into account the specific layout of antennas of the radarsubsystem or may be antenna layout agnostic. Through a training process,a beam-steering direction in which the most vital-signal relatedmovement occurs may be identified by the contactless sleep trackingdevice. The weights associated with this direction may be applied duringsleep tracking such that digital beam-steering is performed to targetthe region where the user is likely located.

A contactless sleep detection device, with or without a beam-steeringmodule, may be used for cough and/or snore detection and attribution forone or more users. Detecting a cough and/or snore based on audio can bea relatively accurate process. However, determining the particularsource of the cough or snore may be challenging. In addition to orinstead of performing sleep tracking, devices detailed herein canfunction as contactless cough detection and attribution devices. (Whilethis document focuses on cough detection and attribution, such systemsand methods can be applied to snoring or other sounds (e.g., talking insleep) by using a detection system configured or trained to detect thedesired sound.

If a contactless cough and/or snore detection and attribution device(“cough attribution device”) is being used to monitor a single user, adetermination can be made as to whether a detected cough was performedby the monitored user. For some embodiments, the cough attributionprocess is at least partially based on FMCW radar signals. For example,the cough may have originated from another person in the vicinity, apet, or audio output by a television or other audio output device. Ifthe cough attribution device is being used to monitor multiple users,the cough may be attributed to one of the monitored multiple users orattributed to none of the multiple users if the cough is determined tohave originated from some source other than the multiple users (again,such as another person in the vicinity, a pet, or audio output by atelevision or other audio output device).

The cough attribution device may incorporate data about a monitored usercoughing into a sleep report provided to the user or cough data may bepresented in a stand-alone report. Cough data for a particular user maybe compiled over an extended period of time (e.g., days, weeks, months)and may allow for cough trend information to be provided to the user,such as an indication that the amount of coughing by the user over theextended period of time is trending up, down, or remaining roughlyconstant.

To perform sleep tracking, cough detection and attribution, and/or otherforms of health monitoring, a setup process may be performed to ensurethat a user has positioned the sleep tracking device or coughattribution device appropriately and that the ambient environment isconfigured in such a way as to permit the device to operate properly.For some embodiments, the setup process includes training the system tolook at or target the user or users typical sleeping location(s) in thebed using beam-steering. A user may request to setup sleep tracking (orother form of health monitoring process) and may take a mock sleepingposition. Using radar, the user may be monitored to determine if theuser is still within a distance range monitored by the sleep trackingdevice. The user may be determined to be static based on a trainedmachine-learning model or, possibly, by detecting the user's breathingto the exclusion of any other significant movements. If the user isdetermined to be present and static, the user may monitored for a periodof time to determine if the user remains present and static for at leasta threshold amount of time (or some other form of determination thatuses a threshold criterion at least partially based on time). If adetermination is made that the user has been classified as present andstatic for a sufficiently long period of time, sleep tracking may beactivated and an indication may be output to the user indicating thatsetup has been successfully performed. By such a setup process beingsuccessfully completed, the device is pointed adequately towards wherethe user sleeps, is at an acceptable distance, and other moving objectshave been removed from the environment. If either the user was notdetermined to be still or, once identified as still, the user was notclassified as remaining within that state for a sufficient period oftime, the setup process may fail and the user may be provided withrecommendations on steps to take to improve the likelihood of itcompleting successfully when setup is again attempted.

Further detail regarding such embodiments and additional embodiments canbe understood in relation to the figures. FIG. 1 illustrates anembodiment of a system 100 for performing contactless sleep detectionand disturbance attribution. System 100 can include: contactless sleeptracking device 101 (“device 101”); network 160; and cloud-based serversystem 170. Device 101 can include: processing system 110; sleep datastorage 118; radar subsystem 120; environmental sensor suite 130;display 140; wireless network interface 150; and speaker 155. Generally,device 101 can include a housing that houses all of the components ofdevice 101. Further detail regarding such a housing, according to someembodiments, is provided in relation to FIG. 3A and FIG. 3B.

Processing system 110 can include one or more processors configured toperform various functions, such as the functions of: radar processingmodule 112; sleep state detection engine 114; and environmental eventcorrelation engine 116. Processing system 110 can include one or morespecial-purpose or general-purpose processors. Such special-purposeprocessors may include processors that are specifically designed toperform the functions detailed herein. Such special-purpose processorsmay be ASICs or FPGAs which are general-purpose components that arephysically and electrically configured to perform the functions detailedherein. Such general-purpose processors may execute special-purposesoftware that is stored using one or more non-transitoryprocessor-readable mediums, such as random access memory (RAM), flashmemory, a hard disk drive (HDD), or a solid state drive (SSD).

Radar subsystem 120 (also referred to as a radar sensor) can be a singleintegrated circuit (IC) that emits, receives, and outputs dataindicative of a received, reflected waveform. The output of radarsubsystem 120 may be analyzed using radar processing module 112 ofprocessing system 110. Further detail regarding radar subsystem 120 andradar processing module 112 is provided in relation to FIG. 2.

Device 101 may include one or more environmental sensors, such as all,one, or some combination of the environmental sensors provided as partof environmental sensor suite 130. Environmental sensor suite 130 caninclude: light sensor 132; microphone 134; temperature sensor 136; andpassive infrared (PIR) sensor 138. In some embodiments, multipleinstances of some or all of these sensors may be present. For instance,in some embodiments, multiple microphones may be present. Light sensor132 may be used for measuring an ambient amount of light present in thegeneral environment of device 101. Microphone 134 may be used formeasuring an ambient noise level present in the general environment ofdevice 101. Temperature sensor 136 may be used for measuring an ambienttemperature of the general environment of device 101. PIR sensor 138 maybe used to detect moving living objects (e.g., persons, pets) within thegeneral environment of device 101. Other types of environmental sensorsare possible. For instance, a camera and/or humidity sensor may beincorporated as part of environmental sensor suite 130. As anotherexample, active infrared sensors may be included. In some embodiments,some data, such as humidity data, may be obtained from a nearby weatherstation that has data available via the Internet. In some embodiments,active acoustic sensing methods, included, but not limited to sonar andultrasound, and including either single or arrayed acoustic sourcesand/or receivers may be implemented. Such arrangements may be used asone or more adjunct sensing modalities incorporated with the othersensors and methods described herein.

In some embodiments, one, some, or all of sensors of environmentalsensor suite 130 may be external device to 101. For instance, one ormore remote environmental sensors may communicate with device 101,either directly (e.g., via a direct wireless communication method, via alow-power mesh network) or indirectly (e.g., through one or more otherdevices via the low-power mesh network, via an access point of anetwork, via a remote server).

Device 101 may include various interfaces. Display 140 can allowprocessing system 110 to present information for viewing by one or moreusers. Wireless network interface 150 can allow for communication usinga wireless local area network (WLAN), such as a WiFi-based network.Speaker 155 can allow for sound, such as synthesized speech, to beoutput. For instance, responses to spoken commands received viamicrophone 134 may be output via speaker 155 and/or display 140. Thespoken commands may be analyzed locally by device 101 or may betransmitted via wireless network interface 150 to cloud-based serversystem 170 for analysis. A response, based on the analysis of the spokencommand, can be sent back to device 101 via wireless network interface150 for output via speaker 155 and/or display 140. Additionally oralternatively, the speaker 155 and microphone 134 may be collectivelyconfigured for active acoustic sensing, including ultrasonic acousticsensing. Additionally or alternatively, other forms of wirelesscommunication may be possible, such as using a low-power wireless meshnetwork radio and protocol (e.g., Thread) to communicate with varioussmart home devices. In some embodiments, a wired network interface, suchas an Ethernet connection, may be used for communication with a network.Further, the evolution of wireless communication to fifth generation(5G) and sixth generation (6G) standards and technologies providesgreater throughput with lower latency which enhances mobile broadbandservices. 5G and 6G technologies also provide new classes of services,over control and data channels, for vehicular networking (V2X), fixedwireless broadband, and the Internet of Things (IoT). Such standards andtechnologies may be used for communication by device 101.

The low-power wireless mesh network radio and protocol may be used forcommunicating with power limited devices. A power-limited device may bean exclusively battery powered device. Such devices may rely exclusivelyon one or more batteries for power and therefore, the amount of powerused for communications may be kept low in order to decrease thefrequency at which the one or more batteries need to be replaced. Insome embodiments, a power-limited device may have the ability tocommunicate via a relatively high power network (e.g., WiFi) and thelow-power mesh network. The power-limited device may infrequently usethe relatively high power network to conserve power. Examples of suchpower-limited devices include environmental sensors (e.g., temperaturesensors, carbon monoxide sensors, smoke sensors, motion sensors,presence detectors) and other forms of remote sensors.

Notably, some embodiments of device 101 do not have any still camera orvideo camera. By not incorporating an on-board camera, users nearby maybe reassured about their privacy. For example, device 101 can typicallybe installed in a user's bedroom. For many reasons, a user would notwant a camera located in such a private space or aimed toward the userwhile the user is sleeping. In other embodiments, device 101 may have acamera, but the camera's lens may be obscured by a mechanical lensshutter. In order to use the camera, the user may be required tophysically open the shutter to allow the camera to have a view of theenvironment of device 101. The user can be assured of privacy from thecamera when the shutter is closed.

Wireless network interface 150 can allow for wireless communication withnetwork 160. Network 160 can include one or more public and/or privatenetworks. Network 160 can include a local wired or wireless network thatis private, such as a home wireless local area network. Network 160 mayalso include a public network, such as the Internet. Network 160 canallow for device 101 to communicate with remotely located cloud-basedserver system 170.

Cloud-based server system 170 can provide device 101 with variousservices. Regarding sleep data, cloud-based server system 170 caninclude processing and storage services for sleep-related data. Whilethe embodiment of FIG. 1 involves processing system 110 performing sleepstate detection and environmental event correlation, in otherembodiments, such functions may be performed by cloud-based serversystem 170. Also, in addition or in alternate to sleep data storage 118being used to store sleep data, sleep-related data may be stored bycloud-based server system 170, such as mapped to a common user accountto which device 101 is linked. If multiple users are monitored, thesleep data may be stored and mapped to a master user account or to thecorresponding users' accounts.

Regardless of whether a single user or multiple users are monitored,each user may be required to provide their informed consent. Suchinformed consent may involve each user consenting to an end useragreement that involves data being used in compliance with HIPAA and/orother generally-accepted security and privacy standards for healthinformation. Periodically, user may be required to renew their consentto collection of sleep data, such as annually. In some embodiments, eachend user may receive a periodic notification, such as via a mobiledevice (e.g., smartphone) that reminds each user that their sleep datais being collected and analyzed and offers each user the option todisable such data collection.

Cloud-based server system 170 may additionally or alternatively provideother cloud-based services. For instance, device 101 may additionallyfunction as a home assistant device. A home assistant device may respondto vocal queries from a user. In response to detecting a vocal triggerphrase being spoken, device 101 may record audio. A stream of the audiomay be transmitted to cloud-based server system 170 for analysis.Cloud-based server system 170 may perform a speech recognition process,use a natural language processing engine to understand the query fromthe user, and provide a response to be output by device 101 assynthesized speech, an output to be presented on display 140, and/or acommand to be executed by device 101 (e.g., raise the volume of device101) or sent to some other smart home device. Further, queries orcommands may be submitted to cloud-based server system 170 via display140, which may be a touchscreen. For instance, device 101 may be used tocontrol various smart home devices or home automation devices. Suchcommands may be sent directly by device 101 to the device to becontrolled or may be sent via cloud-based server system 170.

Based on data output by radar processing module 112, sleep statedetection engine 114 may be used to determine whether a user is likelyasleep or awake. Sleep state detection engine 114 may progress through astate machine, such as detailed in relation to FIG. 5, or may use thestate identified using such a state machine to determine whether theuser is likely awake or asleep. For example, if a user is determined tobe in bed and still for at least a period of time, the user may beidentified as asleep. The output of sleep state detection engine 114 maybe used by environmental event correlation engine 116. Environmentalevent correlation engine 116 may analyze data received fromenvironmental sensor suite 130. Data from each environmental sensordevice may be monitored for: 1) an increase of the environmentalcondition above a fixed defined threshold (or some other form ofdetermination that uses a threshold criterion); and/or 2) an increase inthe environmental condition by at least a predefined amount orpercentage. Alternatively, some other form of threshold criterion may beused to analyze changes in the environmental condition. As an example,data indicating the light level in the ambient environment may becontinuously or periodically output by light sensor 132. Environmentalevent correlation engine 116 may determine whether: 1) the ambientamount of lighting has increased from below a fixed defined threshold toabove the fixed defined threshold (or some other form of determinationthat uses a threshold criterion at least partially based on lighting);and/or 2) the ambient amount of lighting has increased by at least apredefined amount of percentage. If options 1, 2, or both occur, it maybe determined that an environmental event has occurred. Thisenvironmental event may be timestamped by environmental eventcorrelation engine 116. Environmental event correlation engine 116 maythen determine whether the user waking can be attributed to theidentified environmental event. Further detail regarding therelationship between environmental events and sleep events is providedin relation to FIG. 6.

FIG. 2A illustrates an embodiment of a sleep tracking system 200A(“system 200A”). System 200A can include radar subsystem 205 (which canrepresent an embodiment of radar subsystem 120); radar processing module210 (which can represent an embodiment of radar processing module 112);and beam-steering module 230.

Radar subsystem 205 may include RF emitter 206, RF receiver 207, andradar processing circuit 208. RF emitter 206 can emit radio waves, suchas in the form of continuous-wave (CW) radar. RF emitter 206 may usefrequency-modulated continuous-wave (FMCW) radar. The FMCW radar mayoperate in a burst mode or continuous sparse-sampling mode. In burstmode, a frame or burst of multiple chirps, with the chirps spaced by arelatively short period of time, may be output by RF emitter 206. Eachframe may be followed by a relatively long amount of time until asubsequent frame. In a continuous sparse-sampling mode, frames or burstsof chirps are not output, rather chirps are output periodically. Thespacing of chirps in the continuous sparse sampling mode may greater induration than the spacing between chirps within a frame of the burstmode. In some embodiments, radar subsystem 205 may operate in a burstmode, but output raw chirp waterfall data for each burst may be combined(e.g., averaged) together to create simulated continuous sparse-sampledchirp waterfall data. In some embodiments, raw waterfall data gatheredin burst mode may be preferable for gesture detection while rawwaterfall data gathered in a continuously-sparse sampling mode may bepreferable for sleep tracking, vital sign detection, and, generally,health monitoring. Gesture detection may be performed by other hardwareor software components that use the output of radar subsystem 205 thatare not illustrated.

RF emitter 206 may include one or more antennas and may transmit at orabout 60 GHz. The frequency of radio waves transmitted may repeatedlysweep from a low to high frequency (or the reverse). The power levelused for transmission may be very low such that radar subsystem 205 hasan effective range of several meters or an even shorter distance.Further detail regarding the radio waves generated and emitted by radarsubsystem 205 are provided in relation to FIG. 2C.

RF receiver 207 includes one or more antennas, distinct from thetransmit antenna(s), and may receive radio wave reflections off ofnearby objects of radio waves emitted by RF emitter 206. The reflectedradio waves may be interpreted by radar processing circuit 208 by mixingthe radio waves being transmitted with the reflected received radiowaves, thereby producing a mixed signal that can be analyzed fordistance. Based on this mixed signal, radar processing circuit 208 mayoutput raw waveform data, which can also be referred to as the raw chirpwaterfall data for analysis by a separate processing entity. Radarsubsystem 205 may be implemented as a single integrated circuit (IC) orradar processing circuit 208 may be a separate component from RF emitter206 and RF receiver 207. In some embodiments, radar subsystem 205 isintegrated as part of device 101 such that RF emitter 206 and RFreceiver 207 are pointing in a same direction as display 140. In otherembodiments, an external device that includes radar subsystem 205 may beconnected with device 101 via wired or wireless communication. Forexample, radar subsystem 205 may be an add-on device to a home assistantdevice.

For radar subsystem 205, if FMCW is used, an unambiguous FMCW range canbe defined. Within this range, a distance to objects can be accuratelydetermined. However, outside of this range, a detected object could beincorrectly interpreted as nearer than an object within the unambiguousrange. This incorrect interpretation can be due to the frequency of themixed signal and the sampling rate of the ADC used by the radarsubsystem to convert the received analog signals to digital signals. Ifthe frequency of the mixed signal is above the Nyquist rate of thesampling of the ADC, the digital data output by the ADC representativeof the reflected radar signal can be incorrectly represented (e.g., as alower frequency indicative of a closer object).

When using device 201 to monitor sleep patterns and vital statistics, auser may be instructed that the user should be the closest person to thedevice 201. However, it may be possible that another person or an animalis present within the bed. It may be necessary to define the unambiguousFMCW range to be far enough, such as two meters, such that both persons(or, approximately the width of the bed) fall within the unambiguousFMCW range of radar subsystem 205. Two meters may be an ideal distancesince this distance is approximately the width of a large commerciallyavailable bed (e.g., a king size bed).

Raw waveform data may be passed from radar subsystem 205 to radarprocessing module 210. The raw waveform data passed to radar processingmodule 210 may include waveform data indicative of continuous sparsereflected chirps due to radar subsystem 205 operating in a continuoussparse sampling mode or due to radar subsystem 205 operating in a burstmode and a conversion process to simulate raw waveform data produced byradar subsystem 205 operating in a continuous sparse sampling mode beingperformed. Processing may be performed to convert burst sampled waveformdata to continuous sparse samples using an averaging process, such aseach reflected group of burst radio waves being represented by a singleaveraged sample. Radar processing module 210 may include one or moreprocessors. Radar processing module 210 may include one or morespecial-purpose or general-purpose processors. Special-purposeprocessors may include processors that are specifically designed toperform the functions detailed herein. Such special-purpose processorsmay be ASICs or FPGAs which are general-purpose components that arephysically and electrically configured to perform the functions detailedherein. General-purpose processors may execute special-purpose softwarethat is stored using one or more non-transitory processor-readablemediums, such as random access memory (RAM), flash memory, a hard diskdrive (HDD), or a solid state drive (SSD). Radar processing module 210may include: movement filter 211; frequency emphasizer 212; range-vitalstransform engine 213; range gating filter 214; spectral summation engine215; and neural network 216. Each of the components of radar processingmodule 210 may be implemented using software, firmware, or asspecialized hardware.

The raw waveform data output by radar subsystem 205 may be received byradar processing module 210 and first processed using movement filter211. In some embodiments, it is important that movement filter 211 isthe initial component used to perform filtering. That is, the processingperformed by radar processing module 210 is not commutative in someembodiments. Typically, vital sign determination and sleep monitoringmay occur when a monitored user is sleeping or attempting to sleep in abed. In such an environment, there may typically be little movement.Such movement may be attributed to the user moving within the bed (e.g.,rolling over while trying to get to sleep or while asleep) and theuser's vital signs, including movement due to breathing and movement dueto the monitored user's heartbeat. In such an environment, a largeportion of emitted radio waves from RF emitter 206 may be reflected bystatic objects in the vicinity of the monitored user, such as amattress, box spring, bed frame, walls, furniture, bedding, etc.Therefore, a large portion of the raw waveform data received from radarsubsystem 205 may be unrelated to user movements and the user's vitalmeasurements.

Movement filter 211 may include a waveform buffer that buffers “chirps”or slices of received raw waveform data. For instance, sampling mayoccur at a rate of 10 Hz. In other embodiments, sampling may be sloweror faster. Movement filter 211 may buffer twenty seconds of received rawwaveform chirps in certain embodiments. In other embodiments, a shorteror longer duration of buffered raw waveform data is buffered. Thisbuffered raw waveform data can be filtered to remove raw waveform dataindicative of stationary objects. That is, for objects that are moving,such as a monitored user's chest, the user's heartbeat and breathingrate will affect the distance and velocity measurements made by radarsubsystem 205 and output to movement filter 211. This movement of theuser will result in “jitter” in the received raw waveform data over thebuffered time period. More specifically, jitter refers to the phaseshifts caused by moving objects reflecting emitted radio waves. Ratherthan using the reflected FMCW radio waves to determine a velocity of themoving objects, the phase shift induced by the motion in the reflectedradio waves can be used to measure vital statistics, including heartrateand breathing rate, as detailed herein.

For stationary objects, such as furniture, a zero phase shift (i.e., nojitter) will be present in the raw waveform data over the buffered timeperiod. Movement filter 211 can subtract out such raw waveform datacorresponding to stationary objects such that motion-indicative rawwaveform data is passed to frequency emphasizer 212 for furtheranalysis. Raw waveform data corresponding to stationary objects may bediscarded or otherwise ignored for the remainder of processing by radarprocessing module 210.

In some embodiments, an infinite impulse response (IIR) filter isincorporated as part of movement filter 211. Specifically, a single-poleIIR filter may be implemented to filter out raw waveform data that isnot indicative of movement. Therefore, the single-pole IIR filter may beimplemented as a high-pass, low-block filter that prevents raw waveformdata indicative of movement below a particular frequency from passingthrough to frequency emphasizer 212. The cut-off frequency may be setbased on known limits to human vital signs. For example, a breathingrate may be expected to be between 10 and 60 breaths per minute.Movement data indicative of a lower frequency than 10 breaths per minutemay be excluded by the filter. In some embodiments, a band-pass filtermay be implemented to exclude raw waveform data indicative of movementat high frequencies that are impossible or improbable for human vitalsigns. For instance, a heartrate, which can be expected to be above abreathing rate, may be unlikely to be above 150 beats per minute for aperson in a resting or near-resting state. Raw waveform data indicativeof a higher frequency may be filtered out by the band pass filter.

In some embodiments, it may be possible to further fine-tune thefrequencies of raw waveform data that movement filter 211 passes tofrequency emphasizer 212. For instance during an initial configurationphase, a user may provide information about the monitored user (e.g.,himself, a child), such as age data. Table 1 indicates typicalrespiratory rates for various ages. Similar data may be present forheartrate. The filter may be configured to exclude data that is outsideof the expected breathing rate, heartrate range, or both.

TABLE 1 Breathing Rate Range Age (breaths per minute) Birth-6 weeks30-60 6 months 25-40 3 years 20-30 6 years 18-25 10 years 17-23 Adults12-18 65-80 years old 12-28 >80 years old 10-30

The vital signs of the monitored user being measured are periodicimpulse events: a user's heartrate may vary over time, but it can beexpected that the user's heart will continue to beat periodically. Thisbeating is not a sinusoidal function, but rather may be understood as animpulse event, more analogous to a square wave having a relatively lowduty cycle that induces motion in the user's body. Similarly, a user'sbreathing rate may vary over time, but breathing is a periodic functionperformed by the user's body that is analogous to sinusoidal function,except that a user's exhale is typically longer than their inhale.Further, at any given time, a particular window of waveform data isbeing analyzed. Since a particular time window of waveform data is beinganalyzed, even a perfect sinusoid within that window can result inspectral leakage in the frequency domain. Frequency components due tothis spectral leakage should be deemphasized.

Frequency emphasizer 212 may work in conjunction with range-vitalstransform engine 213 to determine the one (e.g., breathing) or two(e.g., breathing plus heartbeat) frequency components of the rawwaveform data. Frequency emphasizer 212 may use frequency windowing,such as a 2D Hamming window (other forms of windowing are possible, suchas a Hann window), to emphasize important frequency components of theraw waveform data and to deemphasize or remove waveform data that isattributable to spectral leakage outside of the defined frequencywindow. Such frequency windowing may decrease the magnitude of rawwaveform data that is likely due to processing artifacts. The use offrequency windowing can help reduce the effects of data-dependentprocessing artifacts while preserving data relevant for being able toseparately determine heartrate and breathing rate.

For a stationary bedside FMCW radar-based monitoring device, which maybe positioned within 1 to 2 meters of the one or more users beingmonitored to detect breathing and heartrate (e.g., using radar asemitted in FIG. 2C), a 2D Hamming window that emphasizes frequencies inthe range of 10 to 60 bpm (0.16 Hz to 1 Hz) for breathing and 30 to 150bpm (0.5 to 2.5 Hz) for heartbeat provide for sufficiently good signalsto make reliable measurements without requiring advance knowledge of thesubject's age or medical history.

Since heartrate and breathing rate are periodic impulse events, thefrequency domain heartrate, and breathing rate may be represented bydifferent fundamental frequencies, but each may have many harmoniccomponents at higher frequencies. One of the primary purposes offrequency emphasizer 212 may be to prevent the frequency ripples ofharmonics of the monitored user's breathing rate from affecting thefrequency measurement of the monitored user's heartrate (or thereverse). While frequency emphasizer 212 may use a 2D Hamming window, itshould be understood that other windowing functions or isolatingfunctions can be used to help isolate frequency ripples of the monitoreduser's breathing rate from the frequency ripples of the monitored user'sheartrate.

Range-vitals transform engine 213 analyzes the received motion-filteredwaveform data to identify and quantify the magnitude of movement atspecific frequencies. More particularly, range-vitals transform engine213 analyzes phase jitter over time to detect relatively small movementsdue to a user's vital signs that have a relatively low frequency, suchas breathing rate and heart rate. The analysis of range-vitals transformengine 213 may assume that the frequency components of the motionwaveform data are sinusoidal. Further, the transform used byrange-vitals transform engine 213 can also identify the distance atwhich the frequency is observed. Frequency, magnitude, and distance canall be determined at least in part because radar subsystem 205 uses anFMCW radar system.

Prior to applying the transform of range-vitals transform engine 213, azero-padding process may be performed by range-vitals transform engine213 to add a number of zeros to the motion-filtered raw waveform data.By performing a zero-padding process, the resolution within thefrequency domain can be increased effectively, allowing for moreaccurate low-rate measurements (e.g., a low heartrate, a low breathingrate). For example, zero-padding can help numerically increaseresolution to detect differences of half a breath per minute compared toa resolution of a breath per minute without zero-padding. In someembodiments, three to four times the number of zeros compared to thebuffered sample size of the raw waveform data may be added. For example,if twenty seconds of buffered raw waveform data are analyzed, sixty toeighty seconds' worth of zero padding may be added to the sample.Specifically, the range of three to four times zero padding of thesample was found to substantially increase resolution while not makingthe transform process overly complex (and, thus, processoruse-intensive).

In order to determine the amount of zero padding to be performed,equations 1-3 may be used. In equation 1, RPM_resolution may ideally beless than 1.

$\begin{matrix}{{RPM}_{resolution} = {60*\frac{chirp\_ rate}{{n\_ fft}{\_ slow}{\_ time}}}} & {{Eq}.1}\end{matrix}$ $\begin{matrix}{{{n\_ FFT}{\_ slow}{\_ time}{\_ min}} = {\left( {{nearest\_ power}{\_ of}\_ 2} \right)\left( {60*{chirp\_ rate}} \right)}} & {{Eq}.2}\end{matrix}$

In some embodiments, a chirp rate (chirp_rate) of 30 Hz may be used.Such a frequency may have sufficient margin from Nyquist limits of theupper limit of breathing rate and heartbeat rate. n_FFT_slow_time_minmay, therefore, be 2048. Given a 20 second window for estimatingrespiration statistics, Equation 3 results in a value of 600.

n_chirps_for_respiration=20*chirp_rate=600  Eq. 3

This value of 600 is smaller than the required vitals-FFT size and makesrange-vitals transform engine 213 perform a 3× to 4× zero padding. Abalance in how much zero padding to perform may be based on increasingthe frequency resolution and associated increases in the amount ofcomputation needed to perform the FFT. A 3× to 4× zero padding has beenfound to provide sufficient resolution for heartrate and breath ratewhile moderating the amount of computation needing to be performed.

Range-vitals transform engine 213 can perform a series of Fouriertransform (FT) to determine the frequency components of the received rawwaveform data output by frequency emphasizer 212. Specifically, a seriesof fast Fourier transform (FFT) may be performed by range-vitalstransform engine 213 to determine the specific frequencies andmagnitudes of waveform data at such frequencies.

Waveform data obtained over a period of time can be expressed inmultiple dimensions. A first dimension (e.g., along the y-axis) canrelate to multiple samples of waveform data from a particular chirp anda second dimension (e.g., along the x-axis) relates to a particularsample index of waveform data gathered across multiple chirps. A thirddimension of data (e.g., along the z-axis) is present indicative of theintensity of the waveform data.

Multiple FFTs may be performed based on the first and second dimensionof the waveform data. FFTs may be performed along each of the first andsecond dimensions: an FFT may be performed for each chirp and an FFT maybe performed for each particular sample index across multiple chirpsthat occurred during the period of time. An FFT performed on waveformdata for a particular reflected chirp can indicate one or morefrequencies, which, in FMCW radar, are indicative of the distances atwhich objects are present that reflected emitted radio waves. An FFTperformed for a particular sample index across multiple chirps canmeasure the frequency of phase jitter across the multiple chirps.Therefore, the FFT of the first dimension can provide the distance atwhich a vital statistic is present and the FFT of the second dimensioncan provide a frequency of the vital statistic. The output of the FFTsperformed across the two dimensions is indicative of: 1) the frequenciesof vital statistics; 2) the ranges at which the vital statistics weremeasured; and 3) the magnitudes of the measured frequencies. In additionto values due to vital statistics being present in the data, noise maybe present that is filtered, such as using spectral summation engine215. The noise may be partially due to heartrate and breathing not beingperfect sinusoidal waves.

To be clear, the transform performed by range-vitals transform engine213 differs from a range-Doppler transform. Rather than analyzingchanges in velocity (as in a range-Doppler transform), periodic changesin phase shift over time are analyzed as part of the range-vitalstransform. The range-vitals transform is tuned to identify smallmovements (e.g., breathing, heart rate) occurring over a relatively longperiod of time by tracking changes in phase, referred to as phasejitter. As previously detailed, zero padding is performed to allow forsufficient resolution for accurate determination of heartrate andbreathing rate.

Range gating filter 214 is used to monitor a defined range of interestand exclude waveform data due to movement beyond the defined range ofinterest. For arrangements detailed herein, the defined range ofinterest may be 0 to 1 meter. In some embodiments, this defined range ofinterest may be different or possibly set by a user (e.g., via atraining or setup process) or by a service provider. In someembodiments, a goal of this arrangement may be to monitor the one personclosest to the device (and exclude or segregate data for any otherperson farther away, such as a person sleeping next to the person beingmonitored). In other embodiments, if both persons are to be monitored,the data may be segregated, as detailed in relation to FIG. 12.Therefore, range-vitals transform engine 213 and range gating filter 214serve to segregate, exclude, or remove movement data attributed toobjects outside of the defined range of interest and sum the energy ofmovement data attributed to objects within the defined range ofinterest. The output of range gating filter 214 may include data thathas a determined range within the permissible range of range gatingfilter 214. The data may further have a frequency dimension and amagnitude. Therefore, the data may possess three dimensions.

Spectral summation engine 215 may receive the output from range gatingfilter 214. Spectral summation engine 215 may function to transfer themeasured energy of harmonic frequencies of the heartrate and breathingrate and sum the harmonic frequency energy onto the fundamentalfrequency's energy. This function can be referred to as a harmonic sumspectrum (HSS). Heartrate and breathing rate are not sinusoidal;therefore, in the frequency domain, harmonics will be present atfrequencies higher than the fundamental frequency of the user'sbreathing rate and the fundamental frequency of the user's heartrate.One of the primary purposes of spectral summation engine 215 is toprevent harmonics of the monitored user's breathing rate from affectingthe frequency measurement of the monitored user's heartrate (or thereverse). The HSS may be performed at the second order by summing theoriginal spectrum with a down-sampled instance (by a factor of two) ofthe spectrum. This process may also be applied at higher order harmonicssuch that their respective spectra are added to the spectrum at thefundamental frequency.

At this stage, for a person in bed who is lying still (with theexception of movement due to breathing and heartrate), it will beexpected that two major frequency peaks will present in the frequencydata. However, if the monitored user is physically moving, such asrolling over in bed, the energy will be significantly distributed acrossthe frequency spectrum (a broader distribution). Such large physicalmovement may manifest itself in the frequency data as being a largenumber of small peaks. If the bed is empty, rather than a person beingpresent, there may be no or almost no frequency components above thenoise floor since movement filter 211 has previously filtered rawwaveform data corresponding to static objects. The distribution andmagnitude of frequency peaks across the spectrum may be used todetermine if the user is likely awake or asleep.

Spectral summation engine 215 may output a feature vector that isindicative of heartrate (e.g., in beats per minute) and breathing rate(e.g., in breaths per minute). The feature vector can indicate frequencyand magnitude. Neural network 216 may be used to determine whether theheartrate and/or breathing rate indicated in the output of the featurevector from spectral summation engine 215 should be considered valid.Therefore, the heartrate and breathing rate output by spectral summationengine 215 may be stored, presented to a user, and/or treated as validbased on the output of neural network 216. Neural network 216 may betrained (e.g., using supervised learning performed using a training setof data) to output one of three states, such as those indicated in Table2 by performing a spectral analysis. Vital statistic data may beconsidered valid when the user is determined to be present and thedetected movement is due to the user's vital signs.

Each state in Table 2 is associated with a different spectral energy andspectral sparsity profile. Spectral energy refers to a summation of theenergy across the frequency spectrum detected due to motion beingpresent within the monitored region. Spectral sparsity representswhether movement tends to be distributed across a wide range offrequencies or clustered at a few specific frequencies. For instance, ifenergy peaks occur at few frequencies, such as when the user's vitalsigns are detected (but not other movement), spectral sparsity is high.However, if peaks (over a threshold value) or some other form ofdetermination based on a threshold criterion at least partially based onmagnitude) occur at many frequencies, spectral sparsity is low.

As an example, motion due to a vital sign, such as a heartbeat, may beindicative of significant movement (e.g., high spectral energy) atspecific frequencies (e.g., high spectral sparsity); motion due to auser moving a limb may also be indicative of significant movement (highspectral energy), but may have low spectral sparsity. The neural networkmay be trained to distinguish between each state based on the spectralenergy profile output by spectral summation engine 215. Therefore,neural network 216 may be provided two features, a first valuerepresenting spectral energy and a second value representing spectralsparsity.

The output of spectral summation engine 215 may be characterized as afeature vector having a first dimension of frequency and a seconddimension of amplitude. The first value representing spectral energy maybe calculated by determining the maximum amplitude present in thefeature vector output by spectral summation engine 215. This maximumamplitude value may be normalized to a value within 0 to 1. The secondvalue representing the spectral sparsity may be calculated bysubtracting the median amplitude of the feature vector from the maximumamplitude. Again here, the calculated sparsity may be normalized to avalue within 0 to 1.

Table 2 represents a generalization of how the features of spectralenergy and spectral sparsity is used as features by the trained neuralnetwork to classify the state of the monitored region.

TABLE 2 State of Monitored Region Spectral Energy Spectral Sparsity Userpresent and vitals-only High High movement User present and moving (limbHigh Low and torso movements) No user present Low Low

The state of the monitored region classified by neural network 216 maybe used in determining the monitored user's sleep state or, moregenerally, whether the user is moving or still within bed. The state ofthe monitored region as determined by the performed classification ofneural network 216 may further be used to determine if the vitalstatistics output by spectral summation engine 215 should be trusted orignored. For accurate vital statistic determination, heartrate andbreathing rate may be identified as likely accurate when neural network216 determines that the user is present and still (i.e., no largephysical movements; however, movement is occurring due to breathingand/or heartbeat). In some embodiments, the vital statistics output byspectral summation engine 215 may be exclusively stored locally (e.g.,to alleviate privacy concerns); in other embodiments, the vitalstatistics output may be transmitted to cloud-based server system 170for remote storage (in alternative or in addition to such data beingstored locally).

Neural network 216 may be initially trained using a large set oftraining data of amplitude and frequency feature vectors that have beenproperly tagged with a classification as mapping the spectral energy andthe spectral sparsity to the corresponding ground-truth state of themonitored region. Alternatively, neural network 216 may be initiallytrained using a large set of training data of amplitude and frequencyfeature vectors that has been properly classified as mapping thecomprising spectral energy and the spectral sparsity pairs each properlytagged to the corresponding ground-truth state of the monitored region.The neural network may be a fully connected neural network that is nottime-dependent. In some embodiments, a machine-learning arrangement,classifier, or form of artificial intelligence other than a neuralnetwork may be used.

In other embodiments, rather than a spectral energy value and a spectralsparsity value being the features used by the neural network, a neuralnetwork, possibly with extra front-end convolutional layers, can betrained to use the output of range gating filter 214 directly. Rather,an embodiment of a convolutional network can analyze the frequency andmagnitude data output by range gating filter 214 to classify the stateof the user. The convolutional neural network may be trained to utilizeoffline training that is based on a set of spectral measurements mappedto the ground truth state of the monitored region prior to system 200Bbeing used by an end user.

The sleep state determined by neural network 216 may be stored, alongwith time data, to sleep data storage 118. The vital statistics outputby spectral summation engine 215 can be stored to a vital statisticdatastore when neural network 216 indicates that the monitored user ispresent and still. Other vital statistic data may be discarded orpossibly flagged to indicate it is less likely to be correct. The datastored to sleep data storage 118 and a vital statistic datastore may bestored locally at device 101. In some embodiments, storage occurs atonly device 101. Such an implementation may help alleviate a concernabout health-related data being transmitted and stored remotely. In someembodiments, the monitored user may elect to have sleep data and vitalstatistic data transmitted via a network interface (e.g., wirelessnetwork interface 150), stored, and analyzed externally, such as bycloud-based server system 170. Storage by cloud-based server system 170may have significant benefits, such as the ability for the user toaccess such data remotely, allow access to a medical provider, orparticipate in research studies. The user may retain the ability todelete or otherwise remove the data from cloud-based server system 170at any time.

In some embodiments, radar processing module 210 may be wholly or partlylocated remotely from device 101. While radar subsystem 205 may need tobe local to the monitored user, the processing of radar processingmodule 210 may be moved to cloud-based server system 170. In otherembodiments, a smart home device that is in local communication (e.g.,via a LAN or WLAN) with device 101 may perform some or all of theprocessing of radar processing module 210. In some embodiments, a localcommunication protocol, such as involving a mesh network, can be used totransmit the raw waveform data to the local device that will beperforming the processing. Such communication protocols can includeWi-Fi, Bluetooth, Thread, or communication protocols of the IEEE 802.11and 802.15.4 families. Similar to the processing, storage of the sleepdata and vital statistic data may occur at cloud-based server system 170or another smart home device in the home at which device 101 is located.In still other embodiments, radar processing module 210 may beincorporated with radar subsystem 205 as a single component or system ofcomponents.

The stored sleep data of sleep data storage 118 and the vital statisticdata may be used to provide the user with short-term and long-termtrends relating to their sleeping patterns, vital statistics, or both bysleep data compilation engine 119. For instance, each morning, graphs,statistics, and trends may be determined by sleep data compilationengine 119 based on data stored to sleep data storage 118 and output fordisplay by sleep data compilation engine 119 via display 140. A graphthat is indicative of sleep data from the previous night and possiblyone or more graphs indicative of breathing rates and heartrate duringthe previous night may be presented. Similar graphs, trends, andstatistics may be output for significantly longer periods of time, suchas weeks, months, years, and even multi-year stretches of time by sleepdata compilation engine 119. Other uses for sleep data and vitalstatistics may be possible. For instance, if certain triggers regardingheartrate, breathing rate, and/or sleeping patterns are triggered, amedical professional may be notified. Additionally or alternatively, anotification may be output to the user indicating that the collecteddata is potentially concerning or is indicative of a healthy person. Insome instances, specific sleep problems may be identified, such as sleepapnea. Sleep data may be output via speaker 155 using synthesized speech(e.g., in response to the user waking, in response to a spoken usercommand, or in response to a user providing input via a touchscreen,such as display 140). Such sleep data may also be representedgraphically and or textually on display 140.

System 200A may additionally include beam-steering module 230.Beam-steering module 230 may include channel weighting engine 231, whichmay be implemented similarly to the components of radar processingmodule 210 using software, firmware, and/or hardware. Beam-steeringmodule 230 is illustrated as separate from radar processing module 210because it processes data received from radar subsystem 205 to emphasizedata received from a particular direction and deemphasize data receivedfrom other directions. Beam-steering module 230 may be implemented usingthe same hardware as radar processing module 210. For instance,beam-steering module 230 may be a software process that modifies theradar data received from radar subsystem 205 prior to movement filter211 being applied. Device 101 may be a surface-top device that isintended to be placed in a particular location, connected with acontinuous power supply (e.g., a household power outlet) and interactedwith via voice and/or a touchscreen. Therefore, radar subsystem 205 mayremain pointed at a portion of the ambient environment for significantperiods of time (e.g., multiple hours, days, weeks, months). Generallyspeaking, beam-steering module 230 may be used to map the environment(e.g., room) in which device 101 is located and steer the sensingdirection of radar subsystem 205 to a zone within the field-of-view ofradar subsystem 205 most likely to have a user present.

Targeting the region within the field-of-view of radar subsystem 205 mayhelp decrease an amount of false negatives and false positives caused bymovement of objects other than a user. Further, targeting can helpcompensate for an angle and location of device 101 relative to where theuser sleeps. (For instance, device 101 may be located on a nightstandthat is at a different height than the user's bed. Additionally oralternatively, radar subsystem 205 device 101 might not be pointeddirectly at the location in the bed where the user sleeps.)

When no user is determined to be present, such as based on low spectralenergy and low spectral density of Table 2, an optimal beam steeringprocess may be performed by channel weighting engine 231 and beamsteering system 232. While no user is present, an analysis can beperformed to determine which directional alignment of radar subsystem205 provides minimal-clutter.

FIG. 2B illustrates an embodiment of a sleep tracking system 200B(“system 200B”) which can perform beam targeting. Beam targetingperformed by using beam-steering module 230 can focus on radarreflections from a region in which a user may be present and ignore orat least decrease the use of radar reflections from objects that causeinterference, such as a nearby wall or large object.

Radar subsystem 240 may contain multiple antennas to receive reflectedradar radio waves. In some embodiments, three antennas may be present.These antennas may be aligned in an “L” pattern, such that two antennasare horizontally orthogonal and two antennas are vertically orthogonalwith one of the antennas being used in both the horizontal arrangementand vertical arrangement. By analyzing the phase difference in receivedradar signals, a weighting may be applied to target the received radarbeam vertically and/or horizontally. In other embodiments, the antennasmay be aligned in a different pattern and/or the beam targeting may beperformed using a single receive antenna and multiple transmit antennasor by both multiple transmit and multiple receive antennas.

Vertical targeting may be performed to compensate for a vertical tilt ofthe device in which system 200B is incorporated. For instance, asdiscussed below in relation to FIG. 3A, the face of contactless sleeptracking device 300 may be tilted with respect to where a user willtypically be sleeping.

Horizontal targeting may be performed to compensate for emitted radarbeing pointed towards an object that causes interference. For instance,if a user's bed headboard is against a wall, the headboard and/or wallmay occupy a significant portion of the field-of-view of radar subsystem120. Radar reflections from the headboard and/or wall are not be usefulin determining data about the user; therefore, it may be beneficial todeemphasize reflections from the wall and/or headboard and emphasizereflections obtained away from the wall and/or headboard. Therefore, thereceive beam may be steered horizontally away from the wall and theheadboard by weighting applied to the received radar signals.

In system 200B, beam-steering module 230 is present to performprocessing on the raw chirp waterfall received from radar subsystem 205before processing is performed by radar processing module 210.Therefore, beam-steering module 230 can function as a preprocessingmodule prior to the analysis of radar processing module 210 and canserve to emphasize regions where one or more users are expected to bepresent. Beam-steering module 230 may be implemented using hardware,software, or firmware; therefore, beam-steering module 230 may beimplemented using the same one or more processors as radar processingmodule 210.

Beam-steering module 230 can include channel weighting engine 231 andbeam steering system 232. Channel weighting engine 231 can be used toperform a training process to determine a series of weightings to beapplied to received radar signals from each antenna prior to thereceived radar signals being mixed together. Channel weighting engine231 may perform a training process when a monitored region is determinedto be empty. During such time, the strength of signals received fromlarge static objects (e.g., walls, headboards) can be analyzed andweightings can be set to steer the beam horizontally (and possiblyvertically) away from such objects. Therefore, the amount of reflectionin a static environment may be minimized for a particular distance range(e.g., up to one meter) from the device by channel weighting engine 231steering the receive radar beam. Such training may also be performedwhen a user is present. That is, the receive beam of radar subsystem 205can be steered to where motion is detected, or specifically, to wherevital signs of a user are present.

The weightings determined by channel weighting engine 231 may be used bybeam steering system 232 to individually apply a weight to the receivedreflected radar signals of each antenna. The received signals from eachantenna may be weighted, then mixed together for processing by radarprocessing module 210. Further detail regarding how various embodimentsof beam-steering module 230 may be implemented are detailed in relationto FIGS. 14-17. Beam-steering module 230 can be used in combination withany other embodiment detailed herein.

FIG. 2C illustrates an embodiment of chirp timing diagram 200C forfrequency modulated continuous wave (FMCW) radar radio waves output by aradar subsystem. Chirp timing diagram 200C is not to scale. Radarsubsystem 205 may generally output radar in the pattern of chirp timingdiagram 200C. Chirp 250 represents a continuous pulse of radio wavesthat sweeps up in frequency from a low frequency to a high frequency. Inother embodiments, individual chirps may continuously sweep down from ahigh frequency to a low frequency, from a low frequency to a highfrequency, and back to a low frequency, or from a high frequency to alow frequency and back to a high frequency. In some embodiments, the lowfrequency is 58 GHz and the high frequency is 63.5 GHz. (For suchfrequencies, the radio waves may be referred to as millimeter waves.) Insome embodiments, the frequencies are between 57 and 64 GHz. The lowfrequency and the high frequency may be varied by embodiment. Forinstance, the low frequency and the high frequency may be between 45 GHzand 80 GHz. The frequencies select may be selected at least in part tocomply with governmental regulation. In some embodiments, each chirpincludes a linear sweep from a low frequency to a high frequency (or thereverse). In other embodiments, an exponential or some other pattern maybe used to sweep the frequency from low to high or high to low.

Chirp 250, which can be representative of all chirps in chirp timingdiagram 200C, may have chirp duration 252 of 128 μs. In otherembodiments, chirp duration 252 may be longer or shorter, such asbetween 50 μs and 1 ms. In some embodiments, a period of time may elapsebefore a subsequent chirp is emitted. Inter-chirp pause 256 may be205.33 μs. In other embodiments, inter-chirp pause 256 may be longer orshorter, such as between 10 μs and 1 ms. In the illustrated embodiment,chirp period 254, which includes chirp 250 and inter-chirp pause 256,may be 333.33 μs. This duration varies based on the selected chirpduration 252 and inter-chirp pause 256.

A number of chirps that are output, separated by inter-chirp pauses maybe referred to as frame 258 or frame 258. Frame 258 may include twentychirps. In other embodiments, the number of chirps in frame 258 may begreater or fewer, such as between 1 and 100. The number of chirpspresent within frame 258 may be determined based upon a maximum amountof power that is desired to be output within a given period of time. TheFCC or other regulatory agency may set a maximum amount of power that ispermissible to be radiated into an environment. For example, a dutycycle requirement may be present that limits the duty cycle to less than10% for any 33 ms time period. In one particular example in which thereare twenty chirps per frame, each chirp can have a duration of 128 us,and each frame being 33.33 ms in duration. The corresponding duty cycleis (20 frames)*(0.128 ms)/(33.33 ms), which is about 7.8%. By limitingthe number of chirps within frame 258 prior to an inter-frame pause, thetotal amount of power output may be limited. In some embodiments, thepeak EIRP (effective isotropically radiated power) may be 13 dBm (20 mW)or less, such as 12.86 dBm (19.05 mW). In other embodiments, the peakEIRP is 15 dBm or less and the duty cycle is 15% or less. In someembodiments, the peak EIRP is 20 dBm or less. That is, at any giventime, the amount of power radiated by the radar subsystem might neverexceed such values. Further, the total power radiated over a period oftime may be limited.

Frames may be transmitted at a frequency of 30 Hz (33.33 ms) as shown bytime period 260. In other embodiments, the frequency may be higher orlower. The frame frequency may be dependent on the number of chirpswithin a frame and the duration of inter-frame pause 262. For instance,the frequency may be between 1 Hz and 50 Hz. In some embodiments, chirpsmay be transmitted continuously, such that the radar subsystem outputs acontinuous stream of chirps interspersed with inter-chirp pauses.Tradeoffs can be made to save on the average power consumed by thedevice due to transmitting chirps and processing received reflections ofchirps. Inter-frame pause 262 represents a period of time when no chirpsare output. In some embodiments, inter-frame pause 262 is significantlylonger than the duration of frame 258. For example, frame 258 may be6.66 ms in duration (with chirp period 254 being 333.33 μs and 20 chirpsper frame). If 33.33 ms occur between frames, inter-frame pause 262 maybe 26.66 ms. In other embodiments, the duration of inter-frame pause 262may be larger or smaller, such as between 15 ms and 40 ms.

In the illustrated embodiment of FIG. 2C, a single frame 258 and thestart of a subsequent frame are illustrated. It should be understoodthat each subsequent frame can be structured similarly to frame 258.Further, the transmission mode of the radar subsystem may be fixed. Thatis, regardless of whether a user is present or not, the time of day, orother factors, chirps may be transmitted according to chirp timingdiagram 200C. Therefore, in some embodiments, the radar subsystem alwaysoperates in a single transmission mode, regardless of the state of theenvironment or the activity attempting to be monitored. A continuoustrain of frames similar to frame 258 may be transmitted while device 101is powered on.

FIG. 3A illustrates an embodiment of a contactless sleep tracking device300 (“device 300”). Device 300 may have a front surface that includes afront transparent screen 340 such that a display is visible. Such adisplay may be a touchscreen. Surrounding front transparent screen 340may be an optically-opaque region, referred to as bezel 330, throughwhich radar subsystem 205 may have a field-of-view of the environment infront of device 300. Cross-section view 400 is detailed in relation toFIG. 4.

For purposes of the immediate following description, the terms verticaland horizontal describe directions relative to the bedroom in general,with vertical referring to a direction perpendicular to the floor andhorizontal referring to a direction parallel to the floor. Since theradar subsystem, which may be an Infineon® BGT60 radar chip, is roughlyplanar and is installed generally parallel to bezel 330 for spatialcompactness of the device as a whole, and since the antennas within theradar chip lie in the plane of the chip, then, without beam targeting, areceive beam of radar subsystem 120 may be pointed in direction 350 thatis generally normal to bezel 330. Due to a departure tilt of bezel 330away from a purely vertical direction, which is provided in someembodiments to be about 25 degrees in order to facilitate easy userinteraction with a touchscreen functionality of the transparent screen340, direction 350 may point upwards from horizontal by departure angle351. Assuming device 300 will typically be installed on a bedsideplatform (e.g., nightstand) that is roughly the same height as the topof a mattress on which a user will sleep, it may be beneficial for thereceive beam of radar subsystem 120 to be targeted in horizontaldirection 352 or an approximately horizontal (e.g., between −5° and 5°from horizontal) direction. Therefore, vertical beam targeting can beused to compensate for departure angle 351 of the portion of device 300in which radar subsystem 120 is present.

FIG. 3B illustrates an exploded view of an embodiment of contactlesssleep tracking device 300. Device 300 can include: display assembly 301;display housing 302; main circuit board 303; neck assembly 304; speakerassembly 305; base plate 306; mesh network communication interface 307;top daughterboard 308; button assembly 309; radar assembly 310;microphone assembly 311; rocker switch bracket 312; rocker switch board313; rocker switch button 314; Wi-Fi assembly 315; power board 316; andpower bracket assembly 317. Device 300 can represent an embodiment ofhow device 101 may be implemented.

Display assembly 301, display housing 302, neck assembly 304, and baseplate 306 may collectively form a housing that houses all of theremaining components of device 300. Display assembly 301 may include anelectronic display, which can be a touchscreen, that presentsinformation to a user. Display assembly 301 may, therefore, include adisplay screen, which can include a metallic plate of the display thatcan serve as a grounding plane. Display assembly 301 may includetransparent portions away from the metallic plate that allow varioussensors a field of view in the general direction in which displayassembly 301 is facing. Display assembly 301 may include an outersurface made of glass or transparent plastic that serves as part of thehousing of device 300.

Display housing 302 may be a plastic or other rigid or semi-rigidmaterial that serves as a housing for display assembly 301. Variouscomponents, such as main circuit board 303; mesh network communicationinterface 307; top daughterboard 308; button assembly 309; radarassembly 310; and microphone assembly 311 may be mounted on displayhousing 302. Mesh network communication interface 307; top daughterboard308; radar assembly 310; and microphone assembly 311 may be connected tomain circuit board 303, using flat wire assemblies. Display housing maybe attached with display assembly 301, using an adhesive.

Mesh network communication interface 307 may include one or moreantennas and may enable communication with a mesh network, such as aThread-based mesh network. Wi-Fi assembly 315 may be located a distancefrom mesh network communication interface 307 to decrease thepossibility of interference. Wi-Fi assembly 315 may enable communicationwith a Wi-Fi based network.

Radar assembly 310, which can include radar subsystem 120 or radarsubsystem 205, may be positioned such that its RF emitter and RFreceiver are away from the metallic plate of display assembly 301 andare located a significant distance from mesh network communicationinterface 307 and Wi-Fi assembly 315. These three components may bearranged in approximately a triangle to increase the distance betweenthe components and decrease interference. For instance, in device 300, adistance of at least 74 mm between Wi-Fi assembly 315 and radar assembly310 may be maintained. A distance of at least 98 mm between mesh networkcommunication interface 307 and radar assembly 310 may be maintained.Additionally, distance between radar assembly 310 and speaker 318 may bedesired to minimize the effect of vibrations on radar assembly 310 thatmay be generated by speaker 318. For instance, for device 300, adistance of at least 79 mm between radar assembly 310 and speaker 318may be maintained. Additionally, distance between the microphones andradar assembly 310 may be desired to minimize any possible interferencefrom the microphones on received radar signals. Top daughterboard 308may include multiple microphones. For instance, at least 12 mm may bemaintained between a closest microphone of top daughterboard 308 andradar assembly 310.

Other components may also be present. A third microphone assembly may bepresent, microphone assembly 311, which may be rear-facing. Microphoneassembly 311 may function in concert with the microphones of topdaughterboard 308 to isolate spoken commands from background noise.Power board 316 may convert power received from an AC power source to DCto power the components of device 300. Power board 316 may be mountedwithin device 300 using power bracket assembly 317. Rocker switchbracket 312, rocker switch board 313, and rocker switch button 314 maybe collectively used to receive user input, such as up/down input. Suchinput may be used, for example, to adjust a volume of sound outputthrough speaker 318. As another user input, button assembly 309 mayinclude a toggle button that a user can actuate. Such a user input maybe used to activate and deactivate all microphones, such as for when theuser desires privacy and/or does not want device 300 to respond to voicecommands.

FIG. 4 illustrates a cross-sectional view of device 300. Screen 401,which may be glass or plastic, may be attached to display housing 302,such as by using adhesive 403. Screen 401 and metallic housing 404 maybe part of display assembly 301. Between radar assembly 310 and screen401 may be air gap 406. Radar assembly 310 may be mounted such that aminimum amount of undesirable reflection is caused by refractive indexdifferences encountered by electromagnetic waves traveling outward fromthe radar assembly 310 and through the front of device 300. Distance 402may be between 2 and 2.3 mm, which corresponds to somewhat less thanhalf of a free-space wavelength of 5 mm at 60 GHz, which may be aboutthe frequency of RF signals output by the RF emitter of radar assembly310. By the distance corresponding to somewhat less (or more) than ahalf wavelength, an anti-cavity is created. If exactly a half wavelengthdistance is used, constructive interference may be present, which can beavoided to prevent unwanted reflected signals from being received.Additionally, a significantly larger or smaller air gap size can be usedto ensure constructive interference does not occur. Adhesive 403 may beconsidered to have little to no effect on radar reflections.

Distance 405 may be at least 1 mm, such as 1.2 mm. The farther radarassembly 310 is separated from metallic housing 404, the lessinterference may be caused by metallic housing 404 on emitted andreceived RF by metallic housing 404. A ground of radar assembly 310 maybe connected with metallic housing 404 such that radar assembly 310 usesmetallic housing 404 as a ground plane.

FIG. 5 illustrates an embodiment of a state machine 500 for determiningwhen a person is sleeping. Based upon data output by radar processingmodule 112, sleep state detection engine 114 may determine whether aperson is sleeping using state machine 500. It should be understood thatin some embodiments, sleep state detection engine 114 is incorporated aspart of the functionality of radar processing module 112 and does notexist as a separate module. State machine 500 may include five possiblesleep states: entering bed state 501; not in bed state 502; motion inbed state 503; no motion in bed state 505; and exiting bed state 504.

If no motion-indicative waveform data is present, this may be indicativethat the user is not in bed. A user who is in bed can be expected toalways be moving in at least small amounts due to their vital signs.Therefore, if zero movement is observed, the user may be judged to be instate 501. Following state 501 being determined, the next possible statethat may be determined is state 502. In state 502, the monitored user isentering bed. Significant user motion may be sensed, such as accordingto Table 2. This may be indicative of a user entering bed and may causethe state to transition from state 501 to state 502.

From state 502, motion may continue to be detected in bed, such as dueto the user rolling around, getting positioned, moving pillows, sheets,and/or blankets, reading a book, etc. State 502 may transition to state503 while such motion continues to be detected. Alternatively, if motionis detected, then zero motion is detected, this may be indicative thatstate 505 has been entered by the monitored user exiting bed. If thiscondition occurs, state 502 may transition to state 505, then back tostate 501. Generally, state 504 may be interpreted as the user beingasleep and state 503 may be interpreted as the user being awake. In someembodiments, more than a threshold amount of time (or some other form ofdetermination that uses a form of threshold criterion at least partiallybased on time) in state 504 is necessary to classify the user as asleepand more than a threshold amount of time (or some other form ofdetermination that uses a form of threshold criterion at least partiallybased on time) in state 503 is necessary to classify the user as awake.For instance, movement in bed of less than five seconds may beinterpreted as the user moving while still asleep if the user waspreviously determined to be asleep. Therefore, if a user transitions tostate 503 from state 504, experiences some number of movement events,then returns to state 504 within less than a duration of time, the usermay be identified as having experienced a “sleep arousal” in which theuser's sleep is disturbed, but the user has not been awoken. Such sleeparousals may be tracked together with or separate data may be maintainedfrom episodes where the user is judged to have fully awoken.

From state 503, the monitored user may be determined to be exiting bedat state 505 and may become motionless at state 504. To be “motionless”at state 504 refers to no large movements being performed by themonitored user, but the user continuing to perform small motions due tovital signs. In some embodiments, only when the monitored user's stateis determined to be state 504 are vital signs treated as accurate and/orstored, recorded, or otherwise used to measure the user's vital signs.Data collected during state 503 and state 504 may be used to determinethe monitored user's general sleep patterns (e.g., how much time tossingand turning, how much quality sleep, when deep sleep occurred, when REMsleep occurred, etc.). After a user enters state 504 for a predefinedperiod of time, the user may be assumed to be asleep until the userexits state 504. When a user initially transitions to state 504, theuser may be required to stay in state 504 for some amount of time, suchas two to five minutes, to be considered asleep. If a user is in state503 for at least a defined period of time, the user may be identified asawake. However, if the user enters state 503 from state 504 for lessthan the defined period of time, and returns to state 504, the user maybe identified as just moving within their sleep and has beencontinuously asleep.

FIG. 6 illustrates timelines of detected sleep states and environmentaldisturbances. Sleep timeline 600 illustrates when a user is determinedto be awake or asleep, such as according to state machine 500. Audiotimeline 610 illustrates when an environmental audio event is detectedby device 101. Light timeline 620 illustrates when an environmentallight event is detected by device 101. In the example of FIG. 6, device101 monitors audio and light. In other embodiments, device 101 maymonitor audio or light. In still other embodiments, one or moreadditional environmental conditions may be monitored, such astemperature or movement of other living things (e.g., using a PIRsensor). A sound event may be detected if: 1) a detected amount of soundin the environment exceeds a fixed sound level threshold (or some otherform of determination that uses a threshold criterion at least partiallybased on sound); or 2) the ambient sound level increases by more than adefined threshold amount or percentage (or some other form ofdetermination that uses a threshold criterion at least partially basedon sound). A light event may be detected if: 1) a detected amount oflight exceeds a fixed light threshold; or 2) the ambient lighting levelincreases by more than a defined threshold amount or percentage (or someother form of determination that uses a threshold criterion at leastpartially based on lighting level). A similar analysis may be performedfor temperature. For movement monitoring, if another living object isdetected moving within the room, an event may be recorded as detected.

A time may be recorded each time a user transitions from being asleep toawake. For each detected audio event and each detected light event, atime may also be recorded and mapped to the environmental event. Todetermine if a sleep event (e.g., a transition from asleep to awake)likely corresponds to an environmental event, if the environmental eventoccurs with a time window around the sleep event, the sleep event may beinterpreted as having been caused by the environmental event. In someembodiments, the time window precedes the sleep event by a fixed amountof time, such as five seconds. In some embodiments, the time windowadditionally trails the sleep event by a fixed amount of time, which canbe shorter than the preceding time, such as by two seconds. In order forthe environmental event to cause the user to awaken, the environmentalevent would logically have to proceed the user waking. However, due tovariances in the detection, processing, and/or analysis of audio, light,temperature, or other factors, it may be accurate to have a trailingtime period after the sleep event during which, if the environmentalevent is determined to have occurred, the environmental event is“blamed” for waking the user. As an example, if significant change intemperature is detected shortly after the user awakes, it may be likelythat it took time for the temperature shift to be detected and the userwas awoken by the temperature change.

In the example of FIG. 6, during time period 601, a user is detected astransitioning from asleep to awake. During time period 601, an audioevent is detected, but no light event. Since the audio event is withinthe defined time period around the sleep event, the user awaking isattributed to the detected audio event. During time period 602, a lightevent is detected but no audio event. Since the light event is withinthe defined time period around the sleep event, the user awaking isattributed to the detected light event. During time period 603, a lightevent and an audio event are detected. Since the light event and audioevent are within the defined time period around the sleep event, theuser awaking is attributed to both the detected light and audio events.During time period 604, no environmental event is detected. Since noenvironmental event is within the defined time period around the sleepevent, the user awaking is not attributed to any environmental event.

The data used to indicate the timelines may be used to provide the userwith a nightly report of sleep. For instance, when requested by the useror at a defined time in the morning, a graphical or textual report maybe presented that indicates: 1) when a user awoke during the night; 2)what environmental events were detected; and 3) instances of waking thatwere attributed to an environmental event.

If multiple users are present within a same bed, movement of a firstuser may be the environmental factor that wakes the second user.Therefore, if a first user is detected as in state 503, and the otheruser is determined to wake within a time window of the first userentering state 503, the first user may be the environmental factor forwaking the second user.

FIG. 7 illustrates an embodiment of raw waveform data or raw chirpwaterfall data in which movement due to vital signs of a user isobservable. Embodiment 700 represents raw waveform data (which can alsobe referred to as raw chirp waterfall data) output by radar subsystem205. Along the x-axis, a chirp index is indicated. This chirp indexindicates an arbitrary identifier for the particular chirp correspondingto the data arranged along the y-axis and the RF intensity data,indicated by the shading. The scale of shading represents normalizedvalues that can be output by the ADC of radar subsystem 205. Along they-axis, a sample index is indicated. For each chirp indicated along thex-axis, a number of samples are measured at a time interval. Forinstance, sixty-four samples may be measured for each chirp. The RFintensity of the reflected radio waves may be measured at each sampleindex.

Embodiment 700 is representative of device 101 being aimed at amonitored sleeping user who is generally still from a distance of lessthan one meter. In embodiment 700, slight “waves” are visible over timein the raw waveform data due to the user's chest and/or abdomen risingand falling thereby effecting the reflection of the radio waves. Thefrequency of these relatively slow movements can be measured over timeto determine a frequency of the user's vital signs. In the illustratedembodiment, the visible waves are caused by the user's breathing patternat approximately 13.5 breaths per minute.

In addition to the visible waves, a significant amount of the RFintensity is due to reflections by static objects. For example, atsample index 64, regardless of the chirp index, the RF intensity remainshigh, possibly due to reflection by a large object, such as a wall. Suchstatic reflections can be filtered out by movement filter 211 prior toother signal processing.

Various methods may be performed using the systems, devices, andarrangements detailed in relation to FIGS. 1-7. FIG. 8 illustrates anembodiment of a method 800 for performing contactless sleep detectionand disturbance attribution. Method 800 may be performed using system100, system 200A, device 300, or some other form of system that cantransmit, receive, and analyze radar, such as an FMCW radar.

At block 805, radio waves are emitted. The radio waves emitted may becontinuous-wave radar, such as FMCW. The radio waves emitted at block805 may be emitted in accordance with the FMCW radar scheme of FIG. 2C.The radio waves emitted may be emitted by RF emitter 206 of radarsubsystem 205. At block 810, reflections of the radio waves arereceived, such as by RF receiver 207 of radar subsystem 205. Thereflections received at block 810 may be reflected off of moving objects(e.g., a person having a heartbeat and breathing) and stationaryobjects. A phase shift may be present in the radio waves reflected by amoving object. For each FMCW chirp emitted at block 805, a number ofsamples may be measured of reflected RF intensity, such as 64 samples.Fewer or greater numbers of samples may be measured in otherembodiments.

At block 815, raw waveform data, which can also be referred to as theraw chirp waterfall data, may be generated based on received reflectedradio waves. A mixed signal generated by mixing the reflected radiowaves with the transmitted radio waves may be indicative of distance anda phase shift. For each of these samples, intensity and phase shift maybe measured. Over time, a window of raw waveform data may be created andstored in a buffer for analysis. Referring to FIG. 2, block 815 may beperformed by radar processing module 210.

At block 820, samples of the waveform data that have been buffered maybe compared. Waveform data that is indicative of static objects (thatis, zero phase shift), which can be defined as objects having movementbelow a particular frequency (or at least a threshold phase shift orsome other form of determination that uses a threshold criterion atleast partially based on phase shift), may be filtered out and discardedsuch that waveform data that is indicative of movement above aparticular frequency is saved for further analysis. Block 220 may beperformed before block 825 to remove a large portion of the waveformdata attributed to static objects and more readily make data attributedto movements of a user detectable.

At block 825, the motion-indicative waveform data may be analyzed. Thisanalysis may be performed to identify and separate data attributable touser motion, heartrate, and breathing rate. Detail regarding how themotion-indicative waveform data may be analyzed is detailed in relationto the components of radar processing module 210. That is, processingusing movement filter 211, frequency emphasizer 212, range-vitalstransform engine 213, range gating filter 214, spectral summation engine215, and neural network 216 may be performed in order to produce data tobe analyzed to determine the user's sleep state and, possibly, theuser's vital statistics.

At block 830, a sleep state of a user may be determined, such as usingthe state machine of FIG. 5. Based upon an output from the radarprocessing module, a sleep state detection engine may determine whethera user is likely asleep (e.g., no significant motion within bed, butvital statistics detected) or awake (e.g., major motion detected withinbed). At block 835, a determination may be made that a user has entereda sleep state based on the analyzed radar data.

While the user is in the sleep state, one or more environmental sensorsmay be active and gathering data that is provided to a processingsystem. The environmental conditions may be monitored to determine if anenvironmental event occurs. Such environmental conditions can includeone or more of: light, sound, temperature, smell, movement, etc. Theseenvironmental conditions may be monitored constantly or periodicallywhile the user is asleep (e.g., following block 830). In someembodiments, the device performing method 800 may perform suchmonitoring constantly regardless of whether the user is detected aspresent (and asleep) or absent.

At block 840, a determination that an environmental event has occurredmay be made based on environmental data obtained from the one or moreenvironmental sensors. At block 840, data from each environmental sensordevice may be monitored for: 1) an increase of the environmentalcondition above a fixed defined threshold (or some other form ofdetermination that uses a threshold criterion); and/or 2) an increase inthe environmental condition by at least a predefined amount orpercentage. If either of these events occurs, an environmental event isidentified as having occurred. An indication of the environmental eventmay be stored in association with a timestamp. At block 845, whileenvironmental conditions are being monitored, the user is determined totransition from sleep to awake. Referring to state machine 500, thisdetermination can involve the state machine being in state 503 for atleast a predetermined amount of time.

At block 850, the user being determined to have entered the awake statefrom the sleep state at block 845 is attributed to the environmentalevent identified at block 840 based on the environmental event occurringwithin a predefined time period of the user waking. The predefined timeperiod may precede the time at which the user awakes, or may span frombefore the user wakes until after the user is identified as awake. Theamount of time preceding the user awaking may be longer than thetrailing time for the time period. In some embodiments, the time periodvaries in duration based on the specific type of environmental event(e.g., a temperature event may involve a longer trailing time than asound event). When an environmental event is attributed with waking theuser, data may be stored indicating the environmental event that causedthe user to wake, when the event occurred, and/or how many times thetype of environmental event has caused the user to awake over someperiod of time (e.g., the last week, the last month).

At block 855, an indication of the user waking one or more times due toone or more environmental events may be output to the user. This outputcan involve a report being presented to the user. The report may be fora particular night or for some other period of time, such as theprevious week. If the number of times a specific type of environmentalevent has woken the user exceeds a defined threshold (or some other formof determination that uses a threshold criterion), the user may bepresented with a recommendation to remedy the environmental event. Insome embodiments, the user may receive an oral report via synthesizedspeech when the user is awake. In some embodiments, the user may beemailed a report periodically about their sleep (e.g., once per week).

While the previous embodiments detailed in relation to the figures areprimarily focused on monitoring the sleep of a single user, these sameconcepts can be applied to multiple users who are sleeping in closeproximity to each other (e.g., within the same bed or within two bedshaving little to no space between them). FIG. 9 illustrates anembodiment 900 of a contactless sleep tracking device monitoringmultiple users. Contactless sleep tracking device 901 (“device 901”) canrepresent an embodiment of contactless sleep tracking device 101 of FIG.1 and/or device 300 that is configured to monitor multiple users over asame time period. In embodiment 900, two users are present within a bedand are both having their sleep monitored by device 901. A differentdistance is present between user 910 and device 901 (distance 911) anduser 920 and device 901 (distance 921).

While separate sleep data may be created and stored by device 901 foreach user, user 910 and user 920 might not be in bed for exactly thesame time period. For example, user 920 may go to bed earlier or laterthan user 910; similarly user 920 may get up from bed in the morningearlier or later than user 910. As a further example, user 910 or user920 may temporarily leave bed in the middle of the night (e.g., for abathroom visit) and subsequently return to bed. Therefore, when a userexits the bed, device 901 can continue to monitor sleeping for the otheruser that remains in bed. Device 901 can track which user has exited bedto ensure that sleep data remains attributed to the correct user despiteone or more exits from and entries to bed.

FIG. 10 illustrates an embodiment of a sleep tracking system 1000 thatcan track multiple users. System 1000 can function similarly to system200A of FIG. 2. More specifically, radar subsystem 205 may be unchangedand beam-steering module 230 may be unchanged. For radar processingmodule 1010, several components may be unchanged from radar processingmodule 210: movement filter 211; frequency emphasizer 212; range-vitalstransform engine 213; and range gating filter 214 may function asdetailed in relation to device 200.

Radar processing module 1010 may additionally include multi-targetsplitter 1011. Multi-target splitter 1011 may serve to: identify thenumber of users present; and map data received from the radar subsystem(which can have been processed using movement filter 211, frequencyemphasized 212, and/or range-vitals transform engine 213) to theassociated user.

As an initial step, multi-target splitter 1011 may compressmulti-dimensional data to fewer dimensions. Data received bymulti-target splitter 1011 may have: a first dimension indicating afrequency of movement; a second dimension of distance from device 901;and/or a third dimension of intensity of movement. One or more of thesedimensions may be eliminated to aid in a clustering process performed bymulti-target splitter 1011. For instance, the dimension of movementfrequency can be removed by using the sample having the greatestmagnitude of movement frequency for a given distance. After suchcompressing, data may have two dimensions: a distance and a magnitude.This data may then be clustered.

Of note, distance can be represented in the multi-dimensional data by asingle value. Therefore, only a distance from system 1000, regardless ofdirection, is tracked. The user, as part of an installation procedure,can be instructed to situate system 1000 to a side of the bed, such ason a nightstand. By system 1000 being to the side of a bed, two (ormore) users lying side-by-side will have their corresponding movementdetected at different distances. In other embodiments, rather than thedata received by multi-target splitter 1011 having three dimensions,four or five dimensions may be present to capture a direction or preciselocation (e.g., 3D coordinates) of movement. By using such a greaternumber of dimensions related to location of movement, the user may befreed to situate system 1000 in any direction relative to the users.

Multi-target splitter 1011 may then be tasked with performing anunsupervised clustering process. The total number of users present isnot known, so the process may need to determine the number of clusters.As previously discussed, while a user may have previously provided datathat indicates two users desire their sleep data monitored, the usersmay enter and/or leave bed at different times. Therefore, at any giventime, the data may need to be analyzed to identify whether one, two, ormore than two users are present. In some embodiments, clustering may berestricted to filing a maximum of two clusters (that is, a restrictionmay be provided that no more than two people are monitored at a time).

To perform unsupervised clustering, multi-target splitter 1011 mayperform a density-based clustering algorithm to the received data (whichhas been reduced by one or more dimensions). Density-based clusteringmay be performed using the density-based spatial clustering ofapplications with a noise (DB SCAN) algorithm. DB SCAN, given a set ofpoints in a space, may group together points that are closely packedtogether (e.g., have many nearby neighbors). It should be understoodthat other forms of clustering besides the DBSCAN algorithm may beperformed. Multi-target splitter 1011 may be initially configured withparameters for the DB SCAN algorithm, such as the minimum number ofpoints required to form a dense region and the size of a point'sneighborhood (typically represented by epsilon).

The output of the clustering process may be an indication of a number ofclusters and a center location of each cluster (or a boundary of thecluster). To separate the clusters, a midpoint location between the twoclusters may be located by multi-target splitter 1011. The midpoint maybe calculated as the midpoint exactly between the locations output bythe clustering algorithm. If three clusters are present (e.g.,indicative of three users), two midpoints may be output, with eachmidpoint being the midpoint between two neighboring users.

Based on the location of the midpoint, a data set may be created foreach user. Therefore, if two clusters are present (indicative of twousers), the data received by multi-target splitter 1011 may be splitinto two datasets based on the location of the midpoint. Each datasetmay subsequently be separately analyzed. Depending on the number ofusers (which is the same as the number of clusters), different instancesof spectral summation engine 1015 (e.g., 1015-1, 1015-2) may be used toindependently analyze the portion of the data output by range gatingfilter 214 mapped to each user. Each spectral summation engine ofspectral summation engines 1015 may function as detailed in relation tospectral summation engine 215 for its respective received portion ofdata from range gating filter 214.

For instance, spectral summation engine 1015-1 may analyze the datadetermined by multi-target splitter 1011 as mapped to a first user andspectral summation engine 1015-2 may analyze the data determined bymulti-target splitter 1011 as mapped to the second user. In someembodiments, there is no overlap in data sent to the separate spectralsummation engines 1015. That is, data from range gating filter 214 issent is separated into two datasets (for two users) and each dataset issent to one of spectral summation engines 1015. If more than two usersare present, a matching number of spectral summation engines 1015 may bepresent to process each data set and the number of datasets createdmatches the number of users detected.

For each user (and, therefore, for each cluster and instance of spectralsummation engine 1015), a separate instance of neural network 1016 (orsome other form of analysis engine) may be implemented (e.g., neuralnetwork 1016-1, 1016-2). Each instance of neural network 1016 mayfunction similarly to neural network 216 for each set of data receivedfrom its corresponding spectral summation engine of spectral summationengines 1015. The output of each of neural networks 1016 may be outputto a corresponding sleep state detection engine of sleep state detectionengines 1018 (e.g., sleep state detection engine 1018-1, sleep statedetection engine 1018-2). Each instance of sleep state detection engine1018 may function similarly to sleep state detection engine 114.Therefore, for each user, their sleep state is monitored independentlybased on the data determined to correspond to that user by multi-targetsplitter 1011. When multiple users are identified, correspondinginstances of a spectral summation engine, neural network, and sleepstate detection engine may be instantiated for each detected user.

The output of each of sleep state detection engine 1018 may be storedand used similarly to what is detailed in relation to FIG. 1. That is,sleep data for each user may be mapped to the appropriate user.Therefore, times at which each user was asleep and awake may be storedsuch that a sleep report can be generated for each user individually(such as similar to sleep timeline 600).

Further, for each user, correlation between waking and environmentalevents may be performed separately. Referring to FIG. 6, a similaranalysis may be performed for each individual user (based on the datamapped to the user by multi-target splitter 1011). For instance, anaudio event may be attributed with waking a first user, but a seconduser may sleep through the audio event (and therefore the audio eventwould be attributed with waking the first user but not be attributedwith waking the second user).

As an additional environmental event that may be monitored (e.g., inaddition to light, sound, temperature, etc.), motion of other users maybe attributed as an environmental factor causing a user to wake. As anexample, movement of a first user (e.g., rolling over in bed whileasleep, waking and getting out of bed) may be sensed using the radarsensor. If, within a sufficiently close time period of the first usermoving, a second user wakes, the first user may be assigned the “blame”for waking the second user. In some embodiments, a determination of whatis sufficiently close can be a defined time period. In this example, ina sleep report for the second user, the first user may be attributed forcausing the second user to wake at a particular time. It may even bepossible for environmental events to be chained together. For instance,an audio event may cause the first user to wake. Motion of the firstuser may then cause the second user to wake. If the two events (theaudio event followed by the second user being woken by the motion of thefirst user) occur within a defined period of time, the waking of thesecond user may be attributed to the audio event or to the combinationof the audio event and the movement of the first user.

FIGS. 11A and 11B illustrate graphs of detected movement at variousdistances, such as detected by device 901. FIG. 11A is a graphicalrepresentation of data that can be received by multi-target splitter1011. In a first dimension (e.g., y-axis), a distance at which movementis detected from the device is indicated. In a second dimension (e.g.,x-axis), a frequency of detected movement may be indicated; in a thirddimension (e.g., z-axis), a magnitude of movement may be indicated. Inthe illustrated graphs, magnitude (along the z-axis) is illustratedusing shading. For example, a heartbeat may have a relatively highmovement frequency but a small magnitude; and breathing may, atapproximately the same distance, have a lower frequency, but a highermagnitude (such as because portions of the user's chest move more due tobreathing than blood pumping).

As can be seen in graph 1110A of FIG. 11A, a single cluster of data ispresent. This arrangement is indicative of a single user being presentwithin the detection range of device 901. However, in graph 1110B ofFIG. 11B, two clusters are present at different distances. The frequencyand magnitude of such data are similar, indicative of, for example, twodifferent users breathing. Therefore, in graph 1110A, a single user ispresent in the detection range of the device 901 but for graph 1110B,two users are present in the detection range of device 901.

FIG. 12 illustrates a graph 1200 of detected movement being split intomultiple targets. Once the multi-target splitter has identified that twoclusters are present and the locations of the two clusters, midpointdistance 1201 between the two locations may be determined. (Aspreviously noted, the number of dimensions of the radar data may havebeen decreased; therefore, while graph 1200 indicates three dimensions,the multi-target splitter may have eliminated one or more dimensions,such as the movement frequency dimension. Therefore, the midpoint may berepresented as a point on the y-axis.)

In graph 1200, motion represented in region 1210 may be attributed to afirst user that is farther away from the device; motion represented inregion 1220 may be attributed to a second user that is closer to thedevice. By monitoring such regions separately, two users can bemonitored. If one of the clusters disappears, it may be indicative ofthe corresponding user exiting bed. The cluster that remains may beattributed to one of the users based on which side of midpoint distance1201 the cluster is present. As long as a single cluster remainspresent, the single cluster may be attributed to the user for which thecluster was initially attributed based on which side of midpointdistance 1201 the cluster was detected. This may remain true even if thecluster migrates over midpoint distance 1201. As an example, consider ifa first user exits bed. The second user may remain sleeping and may rollover to the center of the bed or even the other side of the bed afterthe first user has exited. The second user's sleep would remain trackedregardless of where the second user migrates to within the bed after thefirst user's exit.

Various methods may be performed to independently track the sleep ofmultiple users within a bed or adjoining beds. FIG. 13 illustrates anembodiment of a method 1300 for performing sleep tracking for multipleusers. Typically, method 1300 may be used to separately monitor twousers. However, it is possible the principles of method 1300 may be usedto monitor more than two users. Method 1300 may be performed using radarprocessing module 1010, which may be incorporated as part of system 200Aor system 200B. Further, such systems may be integrated as part of asingle device, such as device 300 of FIGS. 3A and 3B.

At block 1305, waveform data may be received and analyzed after someamount of processing has been performed on the raw chirp waterfall datathat has been received from a source. For example, referring to system200A, radar subsystem 205 may output raw chirp waterfall data that isprocessed using movement filter 211, frequency emphasizer 212,range-vitals transform engine 213, and range gating filter 214. Theprocessed waveform or waterfall data output by range gating filter 214may be graphically represented similarly to the data of graphs 1100A and1100B. That is, the data output by range gating filter 214 may have afirst dimension representative of distance, a second dimensionrepresentative of frequency, and a third dimension representative ofmagnitude. In order to separate data for multiple users, this data mayinitially have one or more dimensions removed. At block 1305, one ormore dimensions of the data may be removed, such as by using a maximumor average value for movement frequency for each distance (e.g., eachdistance range). By performing such a conversion, the frequencydimension may be eliminated, and the data may now only have distance andmagnitude components for the analysis of block 1310.

At block 1310, a clustering process may be performed. The clustering maybe performed on the data that has been reduced in dimensions from block1305. The clustering of block 1310 can be understood as an unsupervisedclustering problem. A key aspect of the clustering may be that thenumber of users present is unknown. For instance, while two users maytypically sleep in a bed, on any given night, just a single usre may bepresent, or the users may enter and exit bed at different times. Adensity-based clustering approach may be taken, such as by using theDBSCAN algorithm or some other algorithm, such as k-means clusteringperformed with progressive counts of assumed clusters, that candetermine a number of clusters present and the location of such clusters(e.g., a center point of each cluster or location of the cluster alongan axis).

At block 1315, based on the number of clusters determined at block 1310,a number of users present may be determined. The number of users maycorrespond to the number of clusters identified at block 1310.Therefore, if two clusters are identified, two users may be identifiedas present. While method 1300 focuses on one or two users being present,it is possible that more than two users may also be identified.

If two users are identified as present, method 1300 may proceed to block1320. At block 1320, a midpoint between the two clusters may bedetermined. The midpoint may be the average of the two locations of theclusters identified at block 1310. In other embodiments, some othermethod for determining a location between the two clusters may beperformed. The point determined at block 1320 can be used to determineto which user data should be attributed.

At block 1325, the processed multidimensional radar waterfall data orwaveform data from block 1305 may then be respectively assigned, ormapped, to each user. In some embodiments, the processedmultidimensional radar waterfall data can be mapped to either user butis not mapped to both. In other embodiments, there can be at least someoverlap in data that is mapped to each user. Therefore, while theprocessed data having the reduced number of dimensions may be used fordetermining the midpoint; further processing may be performed using themultidimensional radar waterfall data from block 1305. At block 1325, afirst point of the multidimensional data is assigned to a first user anda second portion of the multidimensional data is assigned to a seconduser based on the midpoint. For instance, the processed multidimensionalradar waterfall data corresponding to a distance greater than themidpoint may be assigned to the second user; the processedmultidimensional radar waterfall data corresponding to a distance lessthan the midpoint may be assigned to the first user.

At block 1330, independent analyses of the multidimensional data mappedto each user may be performed. The independent analyses may be performedto independently determine the sleep state of each user and determineany environmental factor that has woken the user (e.g., as detailed inrelation to FIG. 6). Referring to FIG. 10, separate instances ofcomponents, such as spectral summation engines 1015 and neural networks1016, may be used to analyze data mapped to each user. Sleep statedetection may then be performed independently for each user based on themapped data.

If, during a subsequent iteration of method 1300 after determining thattwo users are present, a determination is made at block 1315 that asingle user is present, method 1300 may proceed to block 1335. Such asituation may occur when two users are sleeping in a same bed and oneuser gets out of bed, either for the day or temporarily. A determinationof which user remains in bed may need to be made to attribute futuresleep events to the still-in-bed user.

At block 1335, a determination may be made as to where the singlecluster is located in relation to a previously determined midpoint fromblock 1320. The most recently determined midpoint from block 1320 may beused or, for example, an average of a number of most-recent midpoints.The user who remains in bed may be identified based on whether thecluster of the single user in bed is closer to the device than themidpoint or farther away from the device than the midpoint. The singlecluster is attributed to the user who was previously identified on thesame side of the midpoint when method 1300 identified two users aspresent. Following this analysis, even if the user rolls or otherwisemoves within bed, the same single user will be attributed with the sleepdata regardless of where the user is located with respect to themidpoint. That is, when a user exits, the midpoint is used to determinewhich user remains; future motion of the user that is still in bed withrespect to the midpoint has diminished significance because it isalready known which user has remained in bed and which has exited.

If the user who exited bed returns, it may be assumed that the user willresume being on the same sides of the bed as previously determined atblock 1325. For instance, if a user on the far side of the bed exited,it may be assumed that when the user returns, they will be the user onthe far side of the bed again.

At block 1340, an analysis may be performed to determine the sleep stateof the single user. A spectral summation engine and neural network maybe used to analyze data mapped to the single user. Sleep state detectioncontinues to be performed for the single user even though another userhas exited bed (or is otherwise no longer detected).

In FIGS. 14-17, a beam-steering module is detailed. The beam-steeringmodules of FIGS. 14 and 16 perform receive-side beam-steering. Thespecific radar subsystem used may reside on a single integrated circuit,such as Infineon's® BGT60 chip. As detailed in relation to FIG. 15, asingle transmit antenna is present and multiple (e.g., 3) receiveantennas are present. In other embodiments, multiple transmit antennasmay be present to perform transmit-side beam-steering in addition to orinstead of receive-side beam-steering.

A beam-steering module may process data provided from each antenna byradar subsystem 205 prior to such data being processed by radarprocessing module 210. A beam-steering module may serve to performbeam-steering, thereby emphasizing reflected radio waves received from adirection in which the user sleeps and deemphasizing reflected radiowaves from other directions, such as static objects (e.g., a wall,headboard, etc.). It should be understood that the term beam-forming maybe used interchangably with the term beam-steering. Beam-steeringmodules, as detailed herein, may be used with any of the detailedembodiments of contactless sleep tracking devices and associatedmethods. Beam-steering modules may function in the analog domain ordigital domain. If the radar subsystem outputs digital data, thebeam-steering module may function fully in the digital domain usingdigital components.

In digital embodiments, functionality of a beam-steering module may beimplemented as software executed by the same one or more processors asradar processing module 210. Alternatively, functionality of abeam-steering module may be implemented by dedicated hardware orincorporated as part of radar subsystem 205.

FIG. 14 illustrates an embodiment 1400 of beam-steering module 1410 fortargeting the direction in which sleep tracking is performed.Beam-steering module 1410 can represent an embodiment of beam-steeringmodule 230. Generally, beam-steering module 1410 may apply a weight toeach antenna data stream received from radar subsystem 205, sum theweighted inputs, and output the combined weighted antenna data stream toradar processing module 210. The weights applied may introduce a delayto the input of a particular antenna, which can be realized by theweight being a complex value. By a delay being introduced to one or moreof the antenna data streams received from the antennas, the antennareceive beam can be effectively steered.

In embodiment 1400, three digital antenna data streams 1420 (1420-1,1420-2, 1420-3) are received from radar subsystem 205 with each digitalantenna data stream corresponding to a separate antenna. Therefore, inthis embodiment, three antennas are present as part of radar subsystem205. In other embodiments, radar subsystem 205 may have fewer (e.g., 2)or greater numbers (e.g., 4, 5, 6, 7, or more) of antennas, each with acorresponding raw antenna data stream output in digital form tobeam-steering module 1410.

Mixers 1430 and combiner 1440 can represent beam steering system 232.Each of antenna data streams 1420 may be input to a separate mixer ofmixers 1430. Mixers 1430 may be digitally implemented and may thereforerepresent software processes. Mixer 1430-1 mixes antenna data stream1420-1 with a weight, represented by a complex value, output by channelweighting engine 231. Mixer 1430-2 mixes antenna data stream 1420-2 witha weight (which may be the same or differ from the weight applied atmixer 1430-1), output by channel weighting engine 231. Mixer 1430-3mixes antenna data stream 1420-3 with a weight (which may be the same ordifferent from each of the weights applied at mixers 1430-1 and 1430-2),output by channel weighting engine 231.

Channel weighting engine 231, which can represent a software process,may perform a training process to determine the values (e.g., complexvalues) representative of the weights that should be output to each ofmixers 1430. In other embodiments, channel weighting engine 231 may beperformed by separate specialized hardware or hardware that isincorporated as part of radar subsystem 205. The digital signalsrepresenting the weights output by channel weighting engine 231 mayeffectively apply a greater or smaller delay to each of antenna datastreams 1420. The weights applied via mixers 1430 may be normalizedto 1. Therefore, the sum of the three weights applied in embodiment 1400may sum to 1.

Beam steering system 232 and beam-steering module 1410 can be used toimplement weighted delay and sum (WDAS) beam-steering via mixers 1430.Equation 4 details how WDAS can be implemented:

{circumflex over (x)}=Σ _(i=1) ^(M) w _(i) *x _(i)=Σ_(i=1) ^(M)(a _(i)*e _(i) ^(−jπθ) ^(i) )*x _(i).  Eq. 4

In Equation 4, w_(i) represents the channel weight, which can be acomplex value to introduce phase delay; x_(i) represents the incomingdigital radar data (e.g., a FMCW radar chirp) from radar subsystem 205;a_(i) represents the complex-valued weights that are responsible forphase-delaying different receive antenna signals with differentmagnitudes. The weightings output by channel weighting engine 231 may bedetermined by performing a least-squares optimization process. The leastsquares optimization process may be performed according to Equation 5.

minimize∥y−Xw∥ ²  Eq. 5

In Equation 5, y represents vectorized data generated using the targetbeam. X represents the antenna data stream data received from radarsubsystem 205; w represents the weights that are to be learned bychannel weighting engine 231. As part of a training process to determinethe most effective weights to target the user, various weights may betested (e.g., in a pattern, randomly) in an attempt to obtain aminimized output of Equation 5. For example, if enough randomizedweights are tested, it can be expected that the minimized output valuecan be obtained within an amount of error. By minimizing the outputvalue according to the least-squares optimization process, the weightscorresponding to the beam direction that most closely targets where theuser is located within the bed may be obtained. These weights may thenbe used for future monitoring of the user. Periodically or occasionally,retraining may be performed to compensate for the user moving within thebed and/or the orientation and/or location of the sleep detection devicebeing changed.

Prior to use, weights may be determined offline to compensate for aknown tilt of the radar subsystem, such as indicated in FIG. 3A andindicated by directions 350 and 352. When a user is present, the optimaldirection is determined for the user, such as by sweeping or randomlyselecting weights. When a user is not present, one or more directions tostationary objects that produce significant reflections can bedetermined such that these one or more directions can be avoided whentargeting a user.

It should be understood that a learning process other than a leastsquares optimization process may be performed by channel weightingengine 231. For instance, in some embodiments, a user may assist in thetraining process by providing an input indicating a direction from thecontactless sleep tracking device to where the user sleeps. In otherembodiments, a different form of automated learning process may beperformed to target the beam at the user.

Channel weighting engine 231 may be triggered to determine weights onsystem 200B being booted or turned on. If motion is detected by system200B, such as via an on-board accelerometer, channel weights may berecalculated.

A summation of the weighted antenna data streams 1435 (e.g., 1435-1,1435-2, and 1435-3), as output by mixers 1430, may be received bycombiner 1440. Combiner 1440 may output a single summed output 1445 toradar processing module 210. By at least one weight (that causes adelay) applied by mixers 1430 differing from the other weights appliedby mixers 1430, the beam is effectively steered in a direction, whichmay have a vertical and/or horizontal component. Processing by radarprocessing module 210 may be performed as detailed in relation to FIGS.2A and 2B.

FIG. 15 illustrates an embodiment of a possible antenna layout of radarsubsystem 1500. Radar subsystem 1500 may represent an embodiment of theintegrated circuit that functions as radar subsystem 205. The entire ICmay have dimensions of 6.5 mm (length 1505) by 5 mm (width 1504). Inother embodiments, the entire IC has a length 1505 by width 1504 ofbetween 7 mm by 7 mm and 4 mm by 4 mm. The illustrated embodiment ofradar subsystem 205 has three receive antennas and one transmit antenna,but other embodiments may have a greater or fewer number of antennas.Radar subsystem 1500 may have receive antennas 1510-1, 1510-2, and1510-3 distributed in an “L” pattern. That is, antennas 1510-1 and1510-2 may be aligned on axis 1501 and antennas 1510-2 and 1510-3 may bealigned on axis 1502 which is perpendicular to axis 1501, as illustratedin FIG. 15. The center of antenna 1510-2 may be located 2.5 mm or lessfrom the center of antenna 1510-1. The center of antenna 1510-2 may belocated 2.5 mm or less from the center of antenna 1510-3.

Transmit antenna 1510-4 may be arranged separately from the L-shapedpattern of the receive antennas 1510-1, 1510-2, and 1510-3. That is, insome embodiments, a center of transmit antenna 1510-4 is not be locatedon an axis with antenna 1510-3 that is parallel to axis 1501. In someembodiments, transmit antenna 1510-4 is on axis 1503 with center ofantenna 1510-1, with axis 1503 being parallel to axis 1502.

Each of antennas 1510 may be hollow rectangular dielectric resonanceantennas (DRAs). Each of antennas 1510 may have a same set ofdimensions. Alternatively, each of receive antennas 1510-1, 1510-2, and1510-3 may have the same dimensions and transmit antenna 1510-4 may varyin dimensions from the receive antennas. In some embodiments, transmitantenna 1510-4 has a larger width, such as 0.2 mm larger, than receiveantennas 1510-1, 1510-2, and 1510-3 but the same length.

In such an arrangement, the phase delay introduced by the appliedweights between the antenna data stream of antenna 1510-1 and the datastream of antenna 1510-2 may affect the vertical direction of thereceive beam and the phase delay introduced by weights between theantenna data stream of antenna 1510-2 and data stream of antenna 1510-3may affect the horizontal direction of the receive beam (assuming theradar subsystem integrated circuit is present within the contactlesssleep tracking device in approximately the same orientation).

In some embodiments, separate antennas are used for transmitting andreceiving. For example, antenna 1510-4 may be used exclusively fortransmitting, while antennas 1510-1, 1510-2, and 1510-3 are usedexclusively for receiving.

Using a radar subsystem in which all the antennas are located on asingle, relatively compact integrated circuit chip, as described, hasbeen found to achieving a good balance of cost savings, reasonableability to perform receive-side beam-steering, and a sufficiently wideantenna pattern in the horizontal plane that is able to encompassescommon bed sizes (e.g., queen, king, full, twin). At the same time, adevice incorporating such a radar subsystem allows it to be placedsufficiently close to a bed (e.g., within 1 m) that can also function asa personal assistant, including alarm clock functionality (which canreplace an alarm clock), home control hub, and/or entertainmenttouchscreen device.

While the beam-steering module of embodiment 1400 does not factor in thearrangement of antennas 1510 with respect to each other, embodiment 1600accommodates the topology of the antenna arrangement of radar subsystem1500. In other embodiments, antennas 1510 may be arranged in a patternother than an “L.”

FIG. 16 illustrates an embodiment 1600 of a beam-steering module fortargeting the direction in which sleep tracking is performed. Inembodiment 1600, the antenna arrangement (i.e., antenna topology) ofradar subsystem 205 is taken into account. By taking the antennatopology into account, more accurate beam-steering may be performed,which can result in more accurate tracking of a user while sleeping inthe user's bed. Antenna 150-1 corresponds to antenna data stream 1420-1,antenna 1510-2 corresponds to antenna data stream 1420-2, and antenna1510-3 corresponds to antenna data stream 1420-3. That is, a phase delayadded between the data streams of antenna 1510-2 of radar subsystem 205and antenna 1510-3 is used for horizontal beam targeting and a phasedelay added between the data streams of antenna 1510-2 and antenna1510-1 is used for vertical beam targeting. Depending on whether thedata stream of antenna 1510-2 is used for vertical or horizontal beamtargeting, a different weight may be applied using separate digitallyimplemented mixers.

As in embodiment 1400, in embodiment 1600, separate digital antenna datastreams 1420 are received from each antenna of radar subsystem 205.Mixers 1630 and combiners 1640 can represent beam steering system 232 ofFIG. 2B. In embodiment 1600, beam-steering module 1610 has four mixers1630 (1630-1, 1630-2, 1630-3, and 1630-4). Similar to embodiment 1400, avalue (e.g., complex value) output by channel weighting engine 231 maybe mixed with each antenna data stream of antenna data streams 1420.However, different weights may be mixed with antenna data stream 1420-2,with two weighted outputs created, to be used separately for horizontaland vertical beam targeting. Antenna data stream 1420-1 may have aweight applied via mixer 1630-1 and may be combined via combiner 1640-1with antenna data stream 1420-2 (which had a weight applied via mixer1630-2). The weights applied at mixers 1630-1 and 1630-2 may sum to anormalized value of 1. Antenna data stream 1420-3 may have a weightapplied via mixer 1630-4 and may be combined via combiner 1640-2 withantenna data stream 1420-2 which had a weight applied via mixer 1630-3.The weights applied at mixers 1630-3 and 1630-4 may sum to a normalizedvalue of 1.

Channel weighting engine 231 may be implemented similarly to asimplemented in embodiment 1400. Channel weighting engine 231 may performa least-squares optimization process or some other optimization processto determine the optimal or near optimal direction of the receive beam.Channel weighting engine 231 may generate four outputs to be used forweighting in embodiment 1600 rather than three outputs as in embodiment1400. Therefore, if a pattern or random set of values output for theweights is used as part of the least-squares optimization process, agreater number of sets of output values in embodiment 1600 may be testedto obtain the optimized set of output values used to set the weights ascompared to embodiment 1400.

Two outputs 1645, output 1645-1 and output 1645-2, may be output toradar processing module 1650. Separate processing may then be performedby radar processing module 1650 for output 1645-1 and output 1645-2. Ata high level, processing by radar processing module 1650 may beperformed on each of outputs 1645 until direction is no longer usedduring processing. In embodiment 1600, separate instances of movementfilters, frequency emphasizers, and range-vital transform engines may beapplied to each of outputs 1645, then the results may be averaged orsummed together. More specifically, output 1645-1 may be output tomovement filter 1651-1, followed by frequency emphasizer 1652-1,followed by range-vitals transform engine 1653-1. Output 1645-2 may beoutput to movement filter 1651-2, followed by frequency emphasizer1652-2, followed by range-vitals transform engine 1653-2. Movementfilters 1651, frequency emphasizers 1652, and range-vital transformengines 1653 may function as detailed in relation to movement filter211, frequency emphasizer 212, and range-vitals transform engine 213 ofFIGS. 2A and 2B. The output of range-vitals transform engine 1653-1 andrange-vitals transform engine 1653-2 may be summed or combined usingcombiner 1654. The output of combiner 1654, which can represent anaverage of the outputs of range-vitals transform engines 1653, may beprocessed by range gating filter 214 and later components as detailed inrelation to FIGS. 2A and 2B.

Since radar processing module 1650 and beam-steering module 1610 may beperformed as a software process executed by a general-purpose processor(or multiple processors), implementing more complex mixing, weights, andmultiple instances of movement filters 1651, frequency emphasizer 1652,and range-vitals transform engine 1653 may require that enoughprocessing capability be available. Therefore, assuming such processingpower is available, no hardware changes to the contactless sleeptracking device may be needed to implement embodiment 1600 instead ofembodiment 1400. In some embodiments, embodiment 1400 may be implementedand if sleep tracking results are inaccurate, embodiment 1600 may beimplemented (or the reverse). Advantageously, in some embodiments inwhich the contactless sleep tracking device 300 comprises a smart-homemanagement device (e.g., a Nest Home Hub) into which the radarfunctionalities are integrated, the smart-home management device being anetwork-connected combination of smart speaker, home assistant, andtouchscreen-based control and/or entertainment hub, improvements toparameter computation methods and even entire radar processingalgorithms can be achieved by in-the-field software updates pushed outas needed by a central cloud server via the Internet.

While the receive-side beam-steering aspects of embodiments 1400 and1600 of FIGS. 14 and 16, respectively, are implemented in the digitaldomain, the functionality of beam-steering modules 1410 and 1610 may beimplemented in the analog domain using analog components. If suchbeam-steering is performed in the analog domain, conversion to thedigital domain may be performed following such analog beam-steeringbeing performed such that digital data is provided to radar processingmodule 210 or 1650.

Various methods can be performed using embodiments of beam-steeringmodules, such as beam-steering modules 1410 and 1610 of FIGS. 14 and 16,respectively. FIG. 17 illustrates an embodiment of a method 1700 fordirectionally targeting sleep tracking (or, possibly, for some otherform of tracking, such as for coughs as detailed in relation to FIGS.18-21). Method 1700 may be performed using systems such as present inembodiments 1400 and 1600. In some of such embodiments, the antennatopology of FIG. 15, or some similar L-shaped topology, may be present.Method 1700 may be performed by such systems which are incorporated aspart of device 300 of FIGS. 3A and 3B.

Method 1700 may be performed in combination with any of the precedingdetailed methods. Therefore, overlap exists in the various blocksperformed as part of the various methods as detailed below.

At block 1705, radio waves are emitted. The radio waves emitted can becontinuous-wave radar, such as FMCW. The raw waveform data passed toradar processing module may include waveform data indicative ofcontinuous sparse reflected chirps due to the radar subsystem operatingin a continuous sparse sampling mode or due to the radar subsystemoperating in a burst mode and a conversion process to simulate rawwaveform data produced by the radar subsystem operating in a continuoussparse sampling mode being performed. The radio waves emitted at block805 may be emitted in accordance with the FMCW radar scheme of FIG. 2C.The radio waves emitted can be emitted by RF emitter 206 of radarsubsystem 205. At block 1710, reflections of the radio waves may bereceived, such as by multiple antennas of RF receiver 207 of radarsubsystem 205. The reflections received at block 1710 may be reflectedoff of moving objects (e.g., a person having a heartbeat and breathing)and stationary objects. For each FMCW chirp emitted at block 1705, anumber of samples may be measured of reflected RF intensity, such as 64samples, at block 1710. Fewer or greater numbers of samples may bemeasured in other embodiments. A phase shift may be present in the radiowaves reflected by a moving object. Blocks 1705 and 1710 may correspondto blocks performed as part of one or more of the other methods detailedherein, such as blocks 805 and 810 of method 800.

At block 1715, raw waveform data, which can also be referred to as rawchirp waterfall data, may be created based on received reflected radiowaves by each antenna. The reflected radio waves may be indicative ofdistance and a phase shift. At a given frequency, such as 10 Hz, anumber of samples may be taken, such as 64 samples. For each of thesesamples, intensity and phase shift data may be present, and may beoutput as a digital antenna data stream, with a separate antenna datastream being present for each antenna used to receive the reflectedradio waves. Further processing may be performed in the digital domain.In other embodiments, the antenna data streams may be output by theradar subsystem as analog data and the weighting process may beperformed in the analog domain. Over time, a window of raw waveform datamay be created and stored in a buffer for analysis. Referring to FIG. 2,block 815 may be performed by radar processing module 210.

At block 1720, a learning process may be performed to determine a weightto apply to each received antenna data stream. As detailed in relationto channel weighting engine 231, various values that function as weightsmay be tested and a most effective set of weights may be determined thatresults in beam-steering that best targets the location of the user. Thevalues applied may be complex and, thus, may function to introduce phasedelay to one or more of the received antenna data streams. Suchintroduced delay can effectively target the receive antenna beam in aparticular direction, which may have a vertical and/or horizontalcomponent.

The learning process performed as part of block 1720 can involve aleast-squares optimization process being performed or some other form ofoptimization. In some embodiments, a particular direction may be lockedor limited for beam-steering purposes. For instance, horizontally, thebeam may be desired to be at 90° to the face of the contactless sleeptracking device, such as shown in FIG. 3A. Alternatively, the beam maybe limited to vary by a limited range from orthogonal to the face of thecontactless sleep tracking device (e.g., 10°). Additionally oralternatively, the values used for weighting may compensate for avertical inclination angle of the display of the contactless sleeptracking device, such as indicated in FIG. 3A in reference to directions350 and 352. Therefore, the values used to determine the optimized anglemay be restricted to within a particular range such that the verticaland/or horizontal direction of the beam stays within particular ranges(e.g., +/−10° horizontally, +2° to −25° vertically).

After the learning process of block 1720 is complete, blocks 1705, 1710,and 1715 continue to be performed such that antenna data streamscontinue to be output by the radar subsystem. At block 1725, the valuesto apply as weights determined at block 1720 may be applied to theantenna data streams to perform beam-steering while performing a sleeptracking process for one or more users. At block 1730, the weightedantenna data streams may be combined, such as by summing the datastreams together. Block 1730 can involve all weighted antenna datastreams being summed together, such as in embodiment 1400, to create asingle output stream. Block 1730 can also include multiple outputstreams being created by summing different groups of weighted antennastreams together, such as in embodiment 1600. As in embodiment 1600, aparticular antenna data stream may be used twice, with different weightsapplied, for use in targeting the receive antenna beam horizontally andvertically.

At block 1735, sleep tracking may be performed using one or morecombined and weighted antenna data streams. In some embodiments, if asingle output is present from block 1730, such as in embodiment 1400,processing may be performed by radar processing module 210 as detailedin relation to movement filter 211, frequency emphasizer 212,range-vitals transform engine 213, range gating filter 214, spectralsummation engine 215, and neural network 216. In other embodiments, ifmore than one output is present from block 1725, such as in embodiment1600, at least some processing of radar processing module 210 may beperformed separate for each weighted and combined antenna data stream.For instance, the processing of a movement filter, frequency emphasizer,and range-vitals transform may be applied separately to each weightedand combined antenna data stream. Following such separate processing,the processing data streams may be averaged together and furtherprocessing as part of the sleep tracking process may be performed, suchas by range gating filter 214, spectral summation engine 215, and neuralnetwork 216, as detailed in relation to FIGS. 2A and 2B. While block1735 is focused on sleep tracking, block 1735 can additionally oralternatively be focused on cough attribution based on user movement asdetailed in relation to FIGS. 18-21.

Embodiments of sleep tracking devices detailed herein can also functionas a cough attribution device. Alternatively, in some embodiments,devices detailed herein do not perform sleep tracking functions butinstead perform cough detection and attribution functions. When sleepdata is presented to a user, cough data may be incorporated therein,such as the time and numbers of coughs for a particular user. Further,cough trends over time may be monitored for a particular user. The usermay be informed how their amount of coughing over time (e.g., days,weeks, months, even years) has increased or decreased.

FIG. 18 illustrates an embodiment of cough detection and attributionsystem 1800 (“cough attribution system 1800”). In some embodiments ofcontactless sleep tracking device 101, the functionality of coughattribution system 1800 is incorporated. Alternatively, coughattribution system 1800 can be implemented in a device that does notperform sleep tracking functions. Cough attribution system 1800, caninclude radar subsystem 205 (which can represent an embodiment of radarsubsystem 120); radar processing module 210 (which can represent anembodiment of radar processing module 112) or radar processing module1010 (which can also represent an embodiment of radar processing module112). In some embodiments, a beam-steering module, such as beam-steeringmodule 230, may be incorporated as part of cough attribution system1800. In other embodiments, a beam-steering module is not present.Advantageously, by virtue of using radar and audio, system 1800 iscapable of performing cough detection and attribution without making anyphysical contact with the monitored user or a bed of the monitored user.

Cough attribution system 1800 can include: microphone 134; radarsubsystem 120 (which can be radar subsystem 205); cough detector 1810;radar processing module 210 (or 1010); cough data storage 1825; coughdecision engine 1820; cough data compilation engine 1830; display 140;wireless network interface 150; and speaker 155. Any component of coughdetector 1810, radar processing module 210 (or 1010), cough data storage1825, cough decision engine 1820, cough data compilation engine 1830,which can represent software processes performed using one or moreprocessors may be executed locally or may be performed remotely using acloud-based server system.

Microphone 134 may continuously receive audio and output data based onthe received audio to cough detector 1810. In some embodiments,microphone 134 is used to monitor an ambient environment for multipleforms of audio, such as a cough, disturbance or a spoken command, whichmay be triggered by a particular keyword or key phrase. In someembodiments, multiple microphones are present as part of the coughattribution device. Audio streams from such separate microphones may becombined together or analyzed separately. In some embodiments, audio isonly monitored for coughs and/or disturbances when a user has beendetected in bed by radar processing module 210 or radar processingmodule 1010, a feature that is advantageous for a bedside monitoringdevice since many users make a significant amount of noise just beforeentering bed, but are generally intending to be more quiet when they getinto bed. Such an automatic entry into the detection mode can obviatethe need for a specific voice command or button-press to start the coughmonitoring process. Alternatively, or as an optional gating overlay forsuch feature, audio monitoring may be required to be explicitlyauthorized each time by the user to be activated. Preferably, the coughattribution system 1800 is configured such that all audio monitoring maybe easily, readily, and verifiably disabled by the user at any time. Forexample, a hardware-based mechanical switch may be provided thatdisables all onboard microphones of cough attribution system 1800.

Cough detector 1810 may be a software-based process that is executed bya processing system, which includes one or more processors, thatdetermines whether a cough is present in the audio stream received frommicrophone 134. Cough detector 1810 may be executed by the sameprocessing system that executed radar processing module 210 or may beperformed by a separate processing system. Cough detector 1810 mayinclude a trained machine learning model that analyzes the receivedaudio stream and outputs an indication of whether a cough is or is notpresent. When a cough is identified, a timestamp may be output alongwith the indication of the cough being present. Additionally oralternatively, to detect sounds other than a cough, a different form ofdetector may be implemented. For instance, in addition or in alternateto cough detector 1810, a snore detector may be implemented.Additionally or alternatively, for detecting talking in a user's sleep,a speech detector may be implemented. Similar components may beimplemented for a user scratching, grinding teeth, flatulence, burping,hiccupping, and/or some other action or bodily function that can beidentified using audio.

The trained machine learning model may include a neural network that wastrained using truth-tagged training data that includes variousidentified coughs and audio samples that do not include coughs. In otherembodiments, the trained machine learning model can use an arrangementother than a neural network to analyze received audio. While coughdetector 1810 may determine whether a cough is present based on theaudio stream received from microphone 134, radar may be used todetermine whether a monitored user is the source of the cough.

After analysis of the audio stream by cough detector 1810, the audiostream received from microphone 134 may be deleted or otherwisediscarded, in which case no audio captured by microphone 134 used forcough analysis would be preserved. Accordingly, even though audiocapture performed via microphone 134 is active for cough detection, auser does not need to have privacy concerns due to the audio beingdiscarded following cough detection being performed on the audio stream.

Radar subsystem 120 may function as detailed in relation FIGS. 1, 2A,and 2B. Raw radar data, based on detected reflected radio waves of anFMCW radar system may be output to radar processing module 210. In someembodiments, the one or more data streams output by radar subsystem 120may first be targeted by a beam-steering module, such as beam-steeringmodule 230, which is detailed in relation to FIGS. 14-17.

Radar processing module 210 may function as detailed in relation toFIGS. 2A and 2B. The output of neural network 216, as previouslydetailed, may be used to determine a state within state machine 500,such as whether a user is present within bed and moving or present inbed and not moving (besides vital signs). Therefore, an output of neuralnetwork 216 (or some other form of classification engine) indicates: 1)whether the user is in bed; and 2) whether the user is moving (more thanjust vital signs) may be output to cough decision engine 1820. In someembodiments, the state output by radar processing module 210 can includea timestamp.

Cough decision engine 1820 may be a software process that is performedby the processing system that performs the functions of radar processingmodule 210 and/or cough detector 1810. (It should be understood that forother detected sounds, such as snoring, a snore decision engine may beused in addition or in alternate to cough decision engine 1820.) Thisprocessing system may have one or more processors. Cough decision engine1820 may analyze an indication of a cough being present received fromcough detector 1810 in combination with an indication of motion of theuser in bed received from radar processing module 210. The timestamps ofthe detected cough from cough detector 1810 and the detected motion inbed may be required to be within a sufficiently small time period ofeach other in order for cough decision engine 1820 to determine that thecough was the cause of the user moving. For instance, if the user movingis detected within a predefined time period, such as within the rangeextending from one second before to three seconds after the cough beingdetected, the cough and movement may be determined to be correlated(that is, the cough caused the movement). Due to the amount ofprocessing time to analyze the radar data compared to the audio data, asufficiently sized time range may be needed to identify the events asrelated. In some embodiments, the time window is +/−1 second. In otherembodiments, the time window is greater (e.g., +/−2 seconds) or smaller(e.g., +/−0.7 seconds).

If a cough is detected by cough detector 1810 but the monitored user isnot attributed with the cough by cough decision engine 1820 based on thedata received from radar processing module 210, cough decision engine1820 may discard information about the cough as unrelated to themonitored user. Alternatively, even if the cough is not identified asoriginating from the monitored user, the cough may be treated as anaudio event, such as detailed in relation to FIG. 6. If the cough (asperformed or output by someone or something else) is identified as anaudio event that caused the monitored user to awake, the cough may betreated as any other audio or may be specifically stored as a coughevent that awoke the monitored user.

Cough decision engine 1820, when a cough is detected and the cough isattributed to the monitored user, may store an indication of the coughand a timestamp of the cough to cough data storage 1825 of coughattribution system 1800. (Additionally or alternatively, if other formsof disturbances, such as snoring are monitored, cough data storage 1825may be used to store such data or a separate data store may be used.) Insome embodiments, the indication of the cough and the timestamp of thecough may be output to a cloud-based server system for storage. In someembodiments, based on the magnitude of the audio analyzed by coughdetector 1810, an indication of the severity of the cough may be stored(e.g., whether it was a small, medium or large cough based upon athreshold-based volume analysis or some other form of determination thatuses a threshold criterion at least partially based on sound volume). Insome embodiments, a continuous string of coughs may be treated as asingle cough event and an indication of a duration of the cough eventmay be stored.

Cough data storage 1825 may be incorporated as part of sleep datastorage 118 or may be separately stored data. For instance, cough datamay be stored in conjunction with sleep data. Cough data storage 1825may represent a non-transitory processor-readable medium, such as amemory.

Cough data compilation engine 1830 may analyze data from cough datastorage 1825 continuously or periodically, such as once per day,possibly when the user wakes in the morning. Cough data compilationengine 1830 may produce a nightly report that outputs data about theuser coughing during the night. The nightly report may includeinformation such as: 1) the number of times the user coughed during thenight; 2) the duration of such coughs; 3) the time of such coughs; 4)whether the coughs awoke the user; and/or 5) the severity of suchcoughs. Such a nightly report may be output using synthesized speech viaspeaker 155 and/or may be represented using text and/or graphicalindicators on display 140. Data from the nightly report may be output toa cloud-based server system via wireless network interface 150 forstorage and/or further analysis. In other embodiments, the raw coughdata from cough decision engine 1820 is output to the cloud-basedstorage system for analysis. For instance, the functionality of coughdata compilation engine 1830 may be performed by a cloud-based serversystem. Cough data compilation engine 1830 may be used to alternativelyor additionally output data about attribution of other sounds, such assnoring, talking, etc.

Cough data compilation engine 1830 may further produce long-term trenddata that is incorporated as part of the nightly report or is part of aseparate long-term trend report. The long-term trend data may be basedon cough data analyzed over a longer period of time than one day or onenight. For instance, the long-term trend data may analyze data over atime period such as: one week, multiple weeks, a month, multiple months,a year, multiple months, or some custom time period, such as a timeperiod when a user identifies that they have an illness. The long-termtrend data may be output to the user as part of the nightly report or ata less-frequent interval, such as once per week, and/or upon userrequest. The long-tern trend data may be used to indicate to the userinformation such as: 1) whether the user's frequency of coughing atnight is increasing, decreasing, or staying the same (e.g., within athreshold number, or some other form of threshold criterion, of coughsof the user's average cough count); 2) whether the user's coughintensity is increasing, decreasing, or staying the same (e.g., within athreshold range of the average intensity or some other form ofdetermination that uses a threshold criterion at least partially basedon intensity); 3) whether the user's cough duration is increasing,decreasing, or staying the same (e.g., within a threshold range of theaverage duration or some other form of determination that uses athreshold criterion at least partially based on cough duration); and/or4) whether coughs are becoming more likely, or becoming less likely, orhave about the same chance of waking the user from their sleep. In someembodiments, the long-term trend data is output when one of the trendsis noteworthy, such as the user's cough frequency has significantlyincreased.

Such long-term trend data may be output using synthesized speech viaspeaker 155 and/or may be represented using text and/or graphicalindicators on display 140. Data from the long-term trend data may beoutput to a cloud-based server system via wireless network interface 150for storage and/or further analysis. In some embodiments, the long-termtrend data for coughing is output with long-term trend data for sleepingof the user. In such embodiments, the functionality of cough datacompilation engine 1830 may be incorporated with a sleep datacompilation engine, such as sleep data compilation engine 119 of FIG. 1.Therefore, cough data may be output with sleep data.

In some embodiments, a single user may be monitored. This may mean thata single user is present in a bed or may mean that the person who isclosest to the cough attribution device is being monitored. However,even if a single person is being monitored, other sources of coughs andcough-like sounds may be nearby, such as other persons, animals (e.g.,pets), wind or weather, passing vehicles, or speakers. FIG. 19illustrates examples of timelines of detected coughs and a user'smovement state for a single monitored user. Timeline 1901 illustrates auser's determined movement state based on radar data generated by radarsubsystem 120 and processed by radar processing module 210. Timeline1902 illustrates when a cough was detected by cough detector 1810 basedon an audio stream from microphone 134. Notably, a cough being presentin timeline 1902 does not necessarily correspond to the user of timeline1901 since the cough may originate from a source other than themonitored user.

During time period 1910, a cough is detected based on the audio streamand motion is detected by radar processing module 210. In such acircumstance, cough decision engine 1820 attributes the monitored userwith having coughed. Similarly, during time period 1940, a cough isdetected based on the audio stream and motion is detected by radarprocessing module 210. Again here, cough decision engine 1820 attributesthe monitored user with having coughed. Data indicative of the cough,cough duration, timestamp of the cough, and cough severity may be storedto cough data storage 1825.

During time period 1920, two coughs are detected. However, no motion ofthe user is detected. Therefore, while a cough may have been present inthe audio, the cough is not attributed to the user and no cough data isstored for the user for those particular instances of coughs. Inaddition to such audio data being indicative of a cough sound havingoriginated from another source, it may be indicative of a false-positiveof a cough having been detected based on the audio. Regardless ofwhether the cough detection is a false positive or if the coughoriginated from a source other than the user, the data corresponding tothat particular “cough” is not stored in relation to the user.

During time period 1930, motion is detected by the user in the bed byradar subsystem 120. This motion represents significant motion that isgreater than movement of the user due to breathing or the user'sheartbeat. However, no cough was detected based on the captured audiostream. Therefore, no cough data is stored for the user for time period1930.

In some embodiments, multiple users may be monitored over a same timeperiod. For example, two users sleeping in a same bed, such as in FIG.9, may each have their sleeping tracked. Additionally or alternatively,each user's coughing may be tracked. If multiple users are having sleepand/or coughing tracked, radar processing module 1010 may be used inlieu of radar processing module 210. In such an embodiment, coughdecision engine 1820 may receive two inputs from radar processing module1010, thereby receiving a separate input for each user. Additionalembodiments for three or more users are also possible by addingadditional instances of the spectral summation engine and the neuralnetwork. Therefore, a separate output may be present for each monitoreduser that indicates whether the user is present in bed and moving orstill. Cough detector 1810 may continue to function as detailed inrelation to cough attribution system 1800.

FIG. 20 illustrates examples of timelines of detected coughs and motionin bed for multiple monitored users. Again here, timeline 1901illustrates what movement state a user is determined to be in based onradar data generated by radar subsystem 120 and processed by radarprocessing module 1010. Timeline 1902 illustrates when a cough wasdetected by cough detector 1810 based on an audio stream from microphone134.

During time period 1910, a cough is detected based on the audio streamand motion is detected by radar processing module 1010 for the firstuser. In such a circumstance, cough decision engine 1820 attributes themonitored first user with having coughed. Similarly, during time period1940, a cough is detected based on the audio stream and motion isdetected by radar processing module 1010 for the first user. Again here,cough decision engine 1820 attributes the monitored first user withhaving coughed. Data indicative of the cough, cough duration, timestampof the cough, and cough severity may be stored to cough data storage1825 as mapped to the first user.

During time period 2010, a cough is detected based on the audio streamand motion is detected by radar processing module 1010 for the secondmonitored user as indicated on timeline 2001. In such a circumstance,cough decision engine 1820 attributes the monitored second user withhaving coughed. Data indicative of the cough, cough duration, timestampof the cough, and cough severity may be stored to cough data storage1825 as mapped to the second monitored user.

During time period 2020, a cough is detected based on the audio streambut no motion of either monitored user is detected that is sufficient toclassify the user as moving within bed. Therefore, the cough of timeperiod 2020 is not mapped to either monitored user. During time period1930, despite the first user moving within bed, because no cough isdetected based on the audio stream, no indication of a cough is storedfor either user. In some circumstances, a user may cough intenselyenough that it causes both users to move (by the coughing user shakingthe bed, which causes the other user to move). In such a circumstance,the user with the greater amount of movement in bed may be attributedwith the cough.

It should be understood that timelines 1901, 1902, and 2001 are examplesonly. The number of coughs detected, whether one or more users aremonitored, and the timing of coughs would vary based on the particularsituation.

Various methods for cough detection and attribution may be performedusing cough attribution system 1800. FIG. 21 illustrates an embodimentof a method 2100 for cough detection and attribution. Method 2100 may beperformed using cough attribution system 1800 or some other similarsystem. Further, cough attribution system 1800 may be incorporated aspart of device 300. For instance, cough attribution system 1800 may beused with radar processing module 210 when a single user is beingmonitored for coughs. System 1800 may be used with radar processingmodule 1010 if two users are being monitored for coughs. Coughattribution system 1800 can, additionally or alternatively, be used inconjunction with a beam-steering module, such as beam-steering module230, beam-steering module 1410, or beam-steering module 1610 in order toperform beam-steering in the direction of where the user or users are inbed. Further, it should be understood that cough detection andattribution can be performed in conjunction with sleep tracking orseparately from sleep tracking. For instance, cough detection andattribution can be performed along with various embodiments of method800, method 1300, and/or method 1700. Alternatively, method 2100 can beperformed as a stand-alone method separate from methods 800, 1300, and1700. If method 2100 is performed as a stand-alone method by device 300,device 300 can be referred to as a contactless cough detection andattribution device. Notably, due to using radar and audio, method 2100can perform cough detection and attribution without any device makingphysical contact with the monitored user or a bed of the monitored user.

In method 2100, two separate processes may be performed in parallel: anaudio monitoring process may be performed in blocks 2105 through 2115and a radar-based movement monitoring process may be performed in blocks2120 through 2140. Both of these processes can be performed repeatedlyand continuously as part of method 2100. At block 2105, audio isdetected using one or more microphones. In some embodiments, such one ormore microphones are located on-board the device performing method 2100or a remote device that has one or more on-board microphones may be usedand an audio stream may be transmitted to a cough detection and trackingdevice for analysis. For instance, the remote device may be a separatehome assistant or smart speaker device. At block 2110, the audio streamoutput by the microphone is analyzed to determine if a cough hasoccurred. Detection of the cough may be performed using a pre-trainedmachine-learning module, which may be a trained neural network. In someembodiments, cough detection is performed exclusively based on audio. Ifa cough is detected, an output may be created that indicates a cough ispresent and a timestamp of the cough.

At block 2115, the audio stream created by the microphone might bedeleted or otherwise discarded. In some embodiments, no portion of theaudio stream is saved (other than an indication of whether or not acough was present in the audio stream). If the device performing method2100 can function as a home assistant, the audio stream can betemporarily stored if the user speaks a keyword or key phrase that isintended to trigger the user's speech to be interpreted as a command orquestion.

At block 2120, radio waves are emitted. The radio waves emitted can becontinuous-wave radar, such as FMCW. The FMCW radar can transmit radiowaves as detailed in relation to FIG. 2C. The radio waves may be emittedby RF emitter 206 of radar subsystem 205. At block 2125, reflections ofthe radio waves may be received, such as by multiple antennas of RFreceiver 207 of radar subsystem 205. The reflections received at block2125 may be reflected off of moving objects (e.g., a person sleeping inbed, a person moving in bed) and stationary objects. Blocks 2120 and2125 can correspond to blocks performed as part of one or more of theother methods detailed herein, such as blocks 805 and 810 of method 800and/or blocks 1705 and 1710.

At block 2130, raw waveform data, which can also be referred to as rawchirp waterfall data, is created based on the received reflected radiowaves and is output by the radar subsystem. Over time, a window of rawwaveform data may be created and stored in a buffer for analysis. Priorto waveform data being provided to a radar processing module forprocessing, waveform data may be processed using a beam-steering moduleto perform a beam-steering process, such as WDAS beam-steering. Forinstance, blocks of method 1700 may be performed at block 2130.

At block 2135, the raw waveform data, which may have been weighted basedon beam-steering, is analyzed at block 2135. Analysis at block 2135 canbe performed in accordance with the processing detailed in relation tomovement filter 211, frequency emphasizer 212, range-vitals transformengine 213, range gating filter 214, spectral summation engine 215, andneural network 216. As detailed in relation to radar processing module1010, if multiple users are being monitored, each user may be mapped toan instance of spectral summation engine and the neural network.

At block 2140, based on the output from neural network 216, a state maybe determined for the user or each user. The state may be determined inaccordance with state machine 500. Therefore, the output of block 2140may be an indication of whether the user is: in bed, and is moving inbed or is motionless (except for vital signs). If multiple users arebeing monitored, the output of block 2140 may be a similar indicationfor each user. A timestamp may be mapped to each determined state thatis output. Following block 2140, the radar process may continue torepeat and monitor for movement.

At block 2145, a determination may be made as to whether the coughoccurred within a predefined time range of the user moving. Timestampsmapped to an indication of the cough and an indication of user movementcan be used to determine if a cough and user movement occurredsufficiently close in time such that the cough can be attributed withthe movement. In some embodiments, in order for block 2145 to bedetermined in the affirmative, a cough is detected based on the audioand, within the predefined time range, the user is determined to bemoving within bed (e.g., state 503). If either no cough is detected orthe monitored user is not moving in bed, block 2145 may be evaluated inthe negative. If multiple users are being monitored, block 2145 may beevaluated in the affirmative for one user and in the negative for theother users. It is also possible that block 2145 is evaluated in thenegative for all users.

At block 2150, in some embodiments, no cough is attributed or recordedindicating that a monitored user has coughed. In some embodiments, ifthe user was detected as having moved in bed (but not coughed) and sleeptracking is being performed, an indication of the user's movement in bedcan be stored for sleep tracking purposes. Following block 2150, theaudio monitoring process and the radar-based movement monitoring processmay continue to be performed to detect and attribute future possiblecoughs and block 2145 may continue to be evaluated in the future.

At block 2155, if block 2145 was determined in the affirmative, anindication that a cough has occurred may be mapped to the monitored userthat coughed and stored. If multiple users are being monitored, theindication may be mapped to the particular monitored user that wasdetermined to have coughed. In some embodiments, additional informationabout the cough may also be stored, such as: the duration of the cough;the number of coughs in a group of rapid coughs (e.g., a coughing fit);and the intensity of the cough. If multiple monitored users are present,while the cough may be mapped to a particular user, the cough may serveas an audio event that has caused the other user to wake as part ofmethod 800. At the end of a night, none, one, several, or many storedcough indications may be present for a particular monitored user. Blocks2105 through 2155 may be repeatedly performed throughout a night while auser is present in bed.

At block 2160, an indication of the cough stored at block 2155 may beoutput. Block 2160 can include a report being output, such as a nightlyreport, which includes cough data for the previous night. Therefore,block 2160 may be performed following cough detection and attributionconcluding for a night, such as in the morning when a user or users areno longer detected as present in bed. The indication of a cough may beincluded in a generated report that indicates: a number of times theparticular user coughed during the night; when the user coughed; howintense the user's coughs were; whether the coughs woke the user; etc.Such a report may be output in response to the user providing input tothe cough detection and attribution device. The user may provide inputvia a touchscreen or the user may speak a command (possibly along with atrigger word or phrase) that requests the nightly report be output. Inother embodiments, the nightly report may automatically be output at aparticular time or when the user is determined to be awake or out of bedafter a time of day (e.g., after 7 AM). The nightly report may be outputusing synthesized speech and/or text and/or graphics on a display of thecough detection and attribution device. If multiple users are beingmonitored, a separate report, or separate data may be output for eachuser. A report that combines data for the multiple users is alsopossible.

Nightly report data may be transmitted to, stored by, and/or analyzed bya remote cloud-based server system. In some embodiments, each coughindication of block 2155 may be transmitted to the cloud-based serversystem for storage and analysis. Alternatively, in some embodiments,data from the generated report may be transmitted to and stored by thecloud-based server system. A user may have the option to prevent anycough-related data from being transmitted to the cloud-based serversystem. In some embodiments, the nightly report may be generated by thecloud-based server system and stored as mapped to a user account suchthat the report can be accessed via one or more other devices (e.g., asmartphone) of the user that have access to the user account.

The cloud-based server system or the cough detection and attributiondevice may also produce long-term trend data using the stored coughindication. Such long-term trend data may be indicative of cough trendsfor the monitored user over a time period such as: multiple nights, aweek, several weeks, a month, several months, a year, several years,etc. The long-term data may indicate: whether the monitored user'sfrequency of coughing is increasing, decreasing, or stayingapproximately constant over the time period; whether the monitoreduser's intensity of coughing is increasing, decreasing, or stayingapproximately constant over the time period; whether the monitoreduser's duration of coughing is increasing, decreasing, or stayingapproximately constant over the time period. Long-term trend data may bemaintained separately for each monitored user.

Similarly to the nightly report data, the user may provide input via atouchscreen or the user may speak a command (along with a trigger wordor phrase) that requests the long-term trend data be output. In otherembodiments, the long-term trend data may be output at a particular timeor when the user is determined to be awake after a defined time (e.g.,after 7 AM). In some embodiments, the long term trend data is output aspart of the nightly report. In some embodiments, the long-term trenddata is output in response to when a change in the long-term trend datais identified as present, such as a change in the frequency at which theuser is coughing has increased over time. The long-term trend data maybe output using synthesized speech and/or text and/or graphics on adisplay of the cough detection and attribution device. If multiple usersare being monitored, separate long-term trend data may be output foreach user or a combined long-term report can be generated.

The nightly report and/or long-term trend data for coughing may beoutput in conjunction with a sleep report for the one or more users. Asingle report may be output that indicates sleep data and coughing datafor a user. For instance, in the morning, a user may view a singlereport that includes data on the user's sleep from the previous nightand data about the user coughing. Long-term sleep and/or cough data maybe incorporated as part of the report. Such reports may be stored usingthe cloud-based server system mapped to a user account to permit accessby the user to the data from a separate device.

In some embodiments, one or more recommendations may be output if theuser's frequency of coughs is relatively high or increasing. Forinstance, if the cough attribution device (or another smart device inthe vicinity) measures humidity using a humidity sensor, arecommendation may be output that the humidity level be raised in theroom where the user sleeps if the measured humidity is below a thresholdvalue (or some other form of determination that uses a thresholdcriterion at least partially based on humidity) on nights that the usertends to cough. Another recommendation may be that the user seek medicalattention, such as in response to a long-term increase in coughing.

To perform sleep tracking, cough detection and attribution, and/or otherforms of health monitoring or tracking, a setup process may be performedto help ensure that a user has positioned the device appropriately andthat the ambient environment is configured in such a way as to permitthe device to operate properly. Without a setup process being performed,the sleep tracking device may be less likely to be aimed correctly inthe direction of where the user sleeps, be located at an acceptabledistance, and/or that moving objects in the vicinity of the user havebeen removed. FIG. 22 illustrates an embodiment of a sleep trackingsystem 2200 that performs a sleep setup process. It should be understoodthat a similar setup process may be performed for a cough attributiondevice or other form of health monitoring or health tracking device.Sleep tracking system 2200 can represent an embodiment of system 200A ofFIG. 2A. Sleep tracking system 2200 can be incorporated as part ofcontactless sleep tracking device 300 or some other stand-alone,contactless health tracking or monitoring device. Sleep tracking system2200 may also be used to perform a setup process ahead of coughdetection and attribution being performed. Prior to sleep tracking orcough detection and attribution being setup, some components of radarprocessing module 210 may be active. Radar processing module 2210represents a subset of the components of radar processing module 210that can be used to perform the setup process. Movement filter 211,frequency emphasizer 212, and range-vitals transform engine 213 canfunction as detailed in relation to system 200A. Training module 2220may use the output of range-vital transform engine 213 from radarprocessing module 2210.

Training module 2220, similar to radar processing module 2210, may beimplemented as software executed using one or more general-purposeprocessors. In other embodiments, dedicated hardware can be used toexecute functions of the components of training module 2220. In someembodiments, training module 2220 may be active prior to a sleeptracking setup process being successfully completed. In suchembodiments, once completed, training module 2220 is deactivated andsystem 2200 can function as system 200A, system 200B, embodiment 1400,embodiment 1600, or system 1800. Alternatively, the system or user mayreinitiate the setup process at some time after a successful setupprocess, such as if the sleep tracking device is having difficultlydetecting a sleeping user, the device is repositioned, periodically, orat some other time in the future.

Training module 2220 can include classifier 2221, consistency monitor2222, and communication output engine 2223. Classifier may receive theoutput of range-vital transform engine 213. Radar subsystem 205 andradar processing module 2210 may operate continuously regardless ofwhether the sleep tracking setup process has been performed. Upon a userproviding input stating that a sleep tracking setup process is to beperformed, training module 2220 may be activated. When training module2220 is activated, classifier 2221 may begin outputting a classificationbased on the data received from radar processing module 2210, such asthe output from range-vitals transform engine 213.

Range-vitals transform engine 213, as previously detailed, analyzes thereceived motion-filtered waveform data to identify and quantify thefrequency, range, and magnitude of movement over time. Classifier 2221receives as its input processed waveform data indicative of themagnitude of different observed frequencies at various distances.

Classifier 2221, prior to performing a classification, may discardwaveform data indicative of movement at distances too close and/or toofar from system 2200. In some embodiments, detected frequencies at adistance of less than 0.25 m or at a distance of greater than 1 m isdiscarded. In other embodiments, the minimum and maximum range distancesmay vary. For instance, the minimum distance can be between 0.1 and 0.5m and/or the maximum distance may be between 0.7 and 1.5 m.

Classifier 2221 can analyze data in chunks of waveform data over time.That is, data from range-vitals transform engine 213, after discardingwaveform data corresponding to movement too close or too far away, maybe aggregated or summed over a time period, such as two seconds. Inother embodiments, shorter or longer durations of time are used tocreate the data chunks, such as chunks that are 0.5 s to 5 s induration. Classifier 2221 may analyze chunks in 1 s strides (a stridebeing the difference in time from when a first chunk beings until thenext chunk begins), therefore, some amount of overlap can be presentbetween chunks, such as 50%. In other embodiments, the stride may begreater or smaller, such as between 0.5 s and 5 s, which alters theamount of overlap.

Classifier 2221 may include a machine learning model, such as a trainedneural network. The machine learning model receives each summed chunk ofdata (which includes frequency, magnitude, and range data) and outputs aclassification selected from multiple possible classifications. In someembodiments, classifier 2221 outputs one of three possibleclassifications. The classification states may be indicative of: 1) nouser present; 2) excessive movement; and 3) static user present. Aclassification of “no user present” corresponds to no user beingdetected. This classification may be indicative of the user beingoutside of the permissible range, the user not being present in theenvironment, or the device that includes system 2200 having its radarsubsystem aimed away from the user. A classification of “excessivemovement” can be indicative that the user is not lying still (e.g., theuser is rolling in bed or otherwise moving) and/or that one or moreother objects are present and moving in the monitored region. Suchobjects may be fans, clocks (e.g., that include a pendulum), movingwater, moving fabric (e.g., curtains moving due to airflow), a plant(e.g., leaves rustling due to airflow), or some other type of movingobject. A classification of “static user present” may be indicative of auser being detected that is not moving. By not moving, the user can belaying still, but still exhibiting vital signs, such as slight movementsdue to the user breathing and the user's heartbeat. Slight musclemovements (e.g., a twitch of a finger or arm, a deep sigh) may betolerated by the machine learning model and classification of “staticuser present” may still be returned.

Classifier 2221 may include a pre-trained neural network model thatanalyzes two or three features received from range-vitals transformengine 213. The features may be selected from the group of frequency,magnitude, and range. It should be understood that in other embodiments,fewer or greater numbers of features may be used to perform aclassification. In other embodiments, fewer or greater numbers ofclassification states may be determined by classifier 2221. Further, inother embodiments, different classification arrangements, both thosethat use other forms of machine-learning and non-machine-learningarrangements, are used. The machine learning model may be trained usinga set of truth-tagged features (e.g., frequency, magnitude, and/orrange) that have been accurately mapped to the desired state for thosefeatures. For example, in a controlled environment a subject may bemonitored and have the features properly classified by a sleep expertbased on whether the subject is motionless, moving, or not present.

In some embodiments, classifier 2221 can use processes other thanmachine-learning models. For instance, classifier 2221 may determinewhether movement is present due to breathing being detected but little,if any, other movement being detected. Therefore, if based on thefrequency and magnitude data received from range-vital transform engine213, a determination is made that a frequency between ten and sixtyhertz is present (or some other range for a specific age as indicated inTable 1) and no other significant amount of movement is observed(besides, possibly, movement due to a heartbeat), a classification of“static user present” may be determined and output. If magnitudes abovea defined threshold (or some other form of determination that uses athreshold criterion at least partially based on magnitude) are observedat multiple frequencies, a determination of “excessive movement” may beoutput. If no magnitude over a defined threshold is detected, adetermination of “no user present” may be output. In other embodiments,rather than using breathing, another vital sign is detected, such as theuser's heartbeat and used to determine a classification. Breathing maybe preferable since a user's chest moves a greater amount due tobreathing than the user's heart beating.

Classifier 2221 may output a single classification at any given timewhile classifier 2221 is active. In some embodiments, if, after adefined time limit (or some other form of time-based criterion) afterthe sleep tracking setup process has begun, such as between five and 20seconds, the classification of “static user present” has not yet beenoutput by classifier 2221, the setup process does not completesuccessfully. In such a situation, communication output engine 2223 canprovide feedback to a user that setup of sleep tracking has failed and,possibly, may provide recommendations on how to improve the chances forsuccess in a future performed setup process. If the classification of“excessive movement” has been identified by classifier 2221, arecommendation may be made for the user to attempt to remove extraneousmovement from the environment, such as moving objects or the userhimself should refrain from moving. If the classification of “no userpresent” is output by classifier 2221 during the failed setup process,the user may be reminded of the distance that the user should be locatedfrom radar subsystem 205 and/or the user may be reminded of how radarsubsystem 205 should be pointed in relation to where the user sleeps.

If classifier 2221 does output a classification of “static user present”prior to the time limit period expiring, this may serve as an indicationthat the user has properly laid in bed, is being detected, and theuser's environment is sufficiently motionless for proper sleep, cough,or health monitoring and tracking. This initial classification of“static user present” can serve as a trigger to begin a consistencycheck performed by consistency monitor 2222. The purpose of consistencymonitor 2222 may be to ensure that the user, while lying in bed, isproperly detected as “static user present” for a sufficient portion oftime such that future monitoring of the user while sleeping will likelyyield usable vital statistic and/or health monitoring data. For example,while “static user present” may have been initially observed byclassifier 2221, this classification may have been transient, such asdue to movement of curtains that have temporarily and sufficientlystopped moving, over the defined time period. In such a situation,despite the temporary classification of “static user present,” it ispossible that excessive movement may be detected due to airflow resumingthat negatively affects accurate monitoring of a user.

Over a period of time, such as five two second chunks, consistencymonitor 2222 can determine if classifier 2221 has output “static userpresent” for a sufficient portion of the time period. For instance, ifthe time period is 10 seconds, in a 10 second window, classifier 2221may be required to output “static user present” for seven seconds, somenumber of chunks, or some other threshold portion of the time period (orsome other form of determination may be performed that uses a thresholdcriterion at least partially based on the amount of time for whichclassifier 2221 is outputting particular state classifications).

If consistency monitor 2222 determines that “static user present” wasoutput by classifier 2221 for at least a threshold amount of the timeperiod (or, again, uses some other form of threshold criterion based atleast partially based on the amount of time in the given states),communication output engine 2223 may indicate that the sleep trackingsetup has completed successfully and that sleep tracking is now properlysetup and activated. A graphical (e.g., via display 140) and/or auditoryoutput (e.g., via speaker 155) may be provided to the user indicatingthat setup has successfully been completed. Whenever the user is presentin bed going forward, the user's sleep may be tracked. Such tracking mayautomatically begin based on the user being detected with the bed. Withsetup successfully completed, training module 2220 may be deactivatedand system 2200 may transition to function as systems 200A, 200B, 1000,and/or 1800.

If consistency monitor 2222 determines that “static user present” wasnot output by classifier 2221 for at least a threshold portion of thetime period (or some other form of threshold criterion-based analysis onthe amount of time in the given state), communication output engine 2223may indicate that the sleep tracking setup was not completedsuccessfully and that sleep tracking has not be activated. Since theuser was previously identified by classifier 2221 as in bed and still, afailure at this point will likely be due to “excessive movement” beingoutput for a significant period of time by classifier 2221, such as dueto the user rolling over, moving, or some other object nearby moving. Agraphical (e.g., via display 140) and/or auditory output (e.g., viaspeaker 155) may be provided to the user by communication output engine2223 indicating that setup failed. A recommendation may be output thatthe user retry sleep tracking setup, remain still in bed, and remove anymoving objects from the environment.

In some embodiments, consistency monitor 2222 may additionally monitorfor variances in the distance at which the user is detected (e.g., basedon breathing being detected). If the variance in distance at which theuser is observed changes more than a distance threshold (or some otherform of determination that uses a threshold criterion at least partiallybased on distance), consistency monitor 2222 may continue to monitor theuser to see if the variance decreases over time. If the variance doesnot decrease by a defined time limit (e.g., 30 seconds) being reached orby some other time-based criterion, the sleep tracking setup process canfail. If the variance in distance at which the user is observed is anacceptable amount, the setup process can be eligible to successfullycomplete.

FIG. 23 illustrates an embodiment of an instructional user interface2300 presented during a sleep setup process. Instructional userinterface 2300 can be presented using a display screen of device 300 (inwhich system 2200 may be incorporated). Instructional user interface2300 can be presented in response to a user providing input (e.g.,making a selection on the touchscreen, speaking a verbal command)indicating that the user desires to perform a sleep or health monitoringsetup process. In some embodiments, “setup” can instead be referred toas “calibration” since the user is potentially moving the device, otherobjects, and/or their own sleeping position in order to successfullycomplete the setup process. Diagram 2301 may be presented as part ofinstructional user interface 2300 to indicate the general positioningand orientation in which device 300 and the user's bed should bearranged. The user may be permitted to skip additional instructions toproceed directly to setup via touch element 2302 or proceed to the nextinstruction user interface via touch element 2303. Page indicator 2304may indicate the number of instruction interfaces and the currentinstruction user interface being presented (in this example, one ofthree) by virtue of a number of elements and which element isemphasized. Written instructions 2305 can indicate how the user shouldarrange the device in relation to the user's bed. Written instructions2305 can also be output via synthesized speech while instructional userinterface 2300 is presented.

FIG. 24 illustrates an embodiment of an instructional user interface2400 presented during a sleep setup process. Instructional userinterface 2400 can be presented using a display screen of device 300 (inwhich system 2200 may be incorporated). Instructional user interface2400 may be presented following a user providing input (e.g., via touchelement 2303, via a voice command) to proceed to the next instructionaluser interface from instructional user interface 2300. Diagram 2401 maybe presented as part of instructional user interface 2400 to indicate,such as in greater detail than diagram 2301, a position and orientationin which device 300, the user's bed, and the user's sleeping positionshould be arranged in relation to each other. The user might bepermitted to skip additional instructions to proceed directly to setupvia touch element 2402 or the user can proceed to the next instructionuser interface via touch element 2403. Page indicator 2404 can indicatethe number of instruction interfaces and the current instruction userinterface (in this case, two of three) being presented by virtue of anumber of elements and which element is emphasized. Written instructions2405 can indicate how the user should position himself relative to thedevice in bed and/or ensure no objects are blocking a path directly fromthe user's chest to the device. Written instructions 2405 may also beoutput via synthesized speech while instructional user interface 2400 ispresented.

FIG. 25 illustrates an embodiment of an instructional user interface2500 presented during a sleep setup process. User interface 2500 can bepresented using a display screen of device 300 (in which system 2200 maybe incorporated). User interface 2500 may be presented following a userproviding input (e.g., via touch element 2403, via a voice command) toproceed to the next instructional user interface from instructional userinterface 2400. The user may be permitted to skip additionalinstructions (and/or the setup process entirely) via touch element 2502or proceed to setup measurements being captured via touch element 2503.Notably, rather than touching touch element 2503, a user may beencouraged or required to use a verbal command to start setup. Using averbal command, such as “setup,” may help allow the user to remainstill, apart from breathing, during setup. That is, when such a verbalcommand is provided, the user does not need to move their arm and handto provide a touch input to trigger the start of the setup measurements.Page indicator 2504 may indicate the number of instruction interfacesand the current instruction user interface being presented by virtue ofa number of elements and which element is emphasized (in this case, thethird of three interfaces). Written instructions 2505 may indicate howthe user should be in bed, alone, and ready to start setup. Writteninstructions 2505 may also be output via synthesized speech while userinterface 2500 is presented.

FIG. 26 illustrates an embodiment of user interface 2600 presentedduring the sleep setup process being performed. User interface 2600 maybe presented using a display screen of device 300 (in which system 2200may be incorporated). User interface 2600 may be presented in responseto a user triggering the setup process, such as using a voice or touchcommand via touch element 2503, from instructional user interface 2500.

A written indication 2601 may be presented that indicates contactlesssetup measurements are being performed. User interface 2600 can includeindicator value 2602 that indicates of how much (e.g., what percent) ofthe setup process has been performed. In the example of user interface2600, 25% is complete. Indicator value 2602 may indicate for eachpercent or at various rounded values, such as every 5%. Visually,animation 2603, or some other animation may be presented to indicate tothe user that the device is functioning and provide a visually pleasingeffect. Animation 2603 may change color over time. Animation 2603 mayhave multiple circular shapes that each have perimeters that fluctuatein a sinusoidal pattern over time. A gradient that decreases inintensity towards the center of animation 2603 from the perimeters ofthe multiple circular shapes may be present. Further, a second gradientmay be present that decreasing in intensity away from the center of thecircular shapes.

In some embodiments of user interface 2600, audio may be output whileuser interface 2600 is presented. The audio may serve to indicate to auser, to whom user interface 2600 might be difficult to see due to theuser laying down in bed, that the setup process is being performed. Theaudio may include music, such as relaxing instrumental music. When themusic ends and an additional sound can be output, such as a ding, fromwhich the user may infer that setup is complete. Additionally oralternatively, synthesized speech may be output that indicates thatsetup is being performed. When setup is complete, synthesized speech maybe output indicating that setup is complete. The next user interfacepresented may be dependent on whether the setup process successfullycompleted or not.

FIG. 27 illustrates an embodiment of a user interface 2700 presentedfollowing a successful setup process. User interface 2700 may bepresented using a display screen of device 300. User interface 2700 maybe indicative that the sleep tracking process (or some other healthmonitoring process) has been successfully completed. User interface 2700may be presented following user interface 2600 if setup was successfullycompleted. Specifically, user interface 2700 may be output bycommunication output engine 2223 when consistency monitor 2222 hassuccessfully completed the consistency check. Diagram 2701 may begraphically indicative of the device being ready. Touch element 2702 mayallow a user to proceed to a next item for setup or return to a homescreen of the device. Notification 2703 may indicate that the device isnow ready for sleep (and/or cough, and/or, more generally, for health)tracking and/or provide one or more tips for good results. Synthesizedspeech may be output that states the contents of notification 2704 whileuser interface 2700 is being presented.

FIG. 28 illustrates an embodiment of user interface 2800 presentedfollowing an unsuccessful sleep setup process. Therefore, user interface2800 may be presented following user interface 2600. User interface 2800may be presented using a display screen of device 300. User interface2800 may be indicative that the sleep tracking setup (or other healthtracking setup) process has not been successfully completed. Userinterface 2800 may be indicative of a “no user present” state has beendetected by classifier 2221. Since the likely cause is that the user istoo close or too far from device 300, the user may receive a distancerecommendation in instructions 2804, such as that the user be less thanan “arm's length” from the device. Synthesized speech may be output thatstates the contents of instructions 2804 while user interface 2800 isbeing presented. Graphical status indicator 2801 may be indicative ofthe device requiring additional input from the user. Touch element 2802may permit a user to retry the setup process. Touch element 2803 maypermit a user to review the instructions presented in instructional userinterfaces 2300-2500.

FIG. 29 illustrates another embodiment of a user interface 2900presented following an unsuccessful sleep setup process. Therefore, userinterface 2900 may be presented following user interface 2600. Userinterface 2900 may be presented using a display screen of device 300.User interface 2900 may be indicative that the sleep tracking setupprocess has not been successfully completed. User interface 2900 may bepresented when a “excessive movement” classification has been detectedby classifier 2221 (and no “static user present” classification has beenoutput) or that consistency monitor 2222 has detected an excess of the“excessive movement” classification over a time period. Since the likelycause is that the user is moving too much or another object nearby ismoving, the user may receive recommendation on how to rectify thesituation in instructions 2904, such as by laying still and removingmoving objects from the general area. Synthesized speech may be outputthat states the contents of instructions 2904 while user interface 2900is being presented. Graphical status indicator 2901 may be indicative ofthe device requiring additional input from the user. Touch element 2902may permit a user to retry the setup process. Touch element 2903 maypermit a user to review the instructions presented in instructional userinterfaces 2300-2500.

For any of interfaces 2300-2900, synthesized speech may be output thatcorresponds to the text presented. Therefore, a user who is lying in bedmay be made aware of the status of the sleep tracking setup processwithout the user physically needing to move their head to see thedisplay screen. The synthesized speech output may match or may differsomewhat from the text presented on the display screen.

It should be understood that for any of interfaces 2300-2900, fewer orgreater numbers of elements may be presented. Further, elements may berearranged or contain varying instructions based on how the deviceshould be setup.

Various methods may be performed using system 2200 and the graphicaluser interfaces of FIGS. 23-29. FIG. 30 illustrates an embodiment ofmethod 3000 for performing an initial setup process of a sleep trackingdevice. However, method 3000 may also be used for performing a setup ofsome other form of health monitoring or tracking device, such as forcough detection and attribution. Method 3000 may be performed usingsystem 2200, which may be implemented on system 100 and/or device 300.Method 3000 may be performed prior to blocks of other methods detailedherein in order to facilitate a setup process before a user performssleep, cough, or some other form of health monitoring or tracking.

At block 3005, a user can provide a request, such as via voice or touchinput, indicating that the user desired to perform a sleep trackingsetup process. In some embodiments, the device may graphically presentan interface that requests the user perform such a process and requeststhe user's consent to continue. The user can be required to provideinput acknowledging that the user does indeed desire sleep tracking tobe setup and the user is willing to participate in the setup process.The user may be given the option of skipping the setup process (butstill enabling sleep, cough, and health tracking). Such an option can bedesirable when an expert, such as a user who has used the device beforeor an installation professional, is using the device and does not needassistance in calibrating the relative positions of the user, device,and the user's sleeping position. The user may be given the option ofdisabling sleep, cough, and/or health tracking and forgo the setupprocess. If selected by the user, method 3000 concludes following block3005 and such features would be disabled.

Following the user requesting sleep tracking setup be performed, block3010 may be performed. At block 3010, instructions may be output, viathe display screen and/or via synthesized speech, that indicates how thedevice should be positioned relative to where the user sleeps, how faraway the user should be located from the device, and that the usershould remove moving objects from the user's immediate environment.

At block 3015, radio waves are emitted by the radar subsystem of thesystem or device performing method 3000. Therefore, no physical contactwith the user is made by any object to perform sleep tracking (or otherform of health monitoring). In some embodiments, radio waves may beginbeing emitted at block 3015; in other embodiments, radio waves may havealready begun being output by the device regardless of whether the sleeptracking setup process is initiated. The radio waves emitted may becontinuous-wave radar, such as FMCW. The radio waves emitted at block805 may be emitted in accordance with the FMCW radar scheme of FIG. 2C.The radio waves are emitted by RF emitter 206 of radar subsystem 205. Atblock 3015, reflections of the radio waves are received, such as bymultiple antennas of RF receiver 207 of radar subsystem 205. Thereflections received at block 3020 are reflected off of moving objects(e.g., a person having a heartbeat and breathing) and stationaryobjects. An output of the radar subsystem based on the receivedreflected radio waves can be processed as detailed in relation tomovement filter 211, frequency emphasizer 212, and range-vitalstransform engine 213. The output of range-vital transform engine 213 mayindicate measured frequencies, frequency magnitudes, and the distancesat which those frequency magnitudes were measured.

At block 3025, a classification may be performed using a trainedclassifier based upon the frequency, frequency magnitude, and distancewaveform data. The classification may be performed as detailed inrelation to classifier 2221, such as using a machine learning model orby determining whether breathing is detected to the exclusion of othersignificant amounts of movement. One of three (or some other number)possible classifications may be output by the classifier. The desiredclassification at this point to continue the setup process is that theuser is present and is static, which would be indicative of the userproperly lying in bed and being motionless, except for movement due tothe user's vital statistics. The classification determined at block 3025may be evaluated at block 3030.

At block 3030, if the user is determined to be present and static basedupon the classification of block 3025, method 3000 may proceed to block3035. If the user is not evaluated to be present and static at block3030, the classification of block 3025 may continue to be performed fora time, such as up until a time limit nor some other limiting time-basedcriterion. If, at any point during that time period, the user isidentified as present and static, method 3000 may proceed to block 3035.Block 3030 may be evaluated in the negative if at no point before thetime limit is reached (or the time-based criterion is reached) is aclassification at block 3025 of the user being present and static made.If block 3030 is evaluated in the negative, method 3000 may proceed toblock 3055.

At block 3055, an indication can be output that sleep tracking setup hasfailed and, possibly, one or more recommendations that the user shouldfollow when trying setup again. If the predominant classification outputat block 3025 is that no user is detected (“user not present”), anindication may be output that the user should re-aim the device towardwhere the user is sleeping and stay within a distance range that ispermissible for the device. If the predominant classification output atblock 3025 is that excessive movement is detected, an indication may beoutput that the user should try moving less and/or remove objects thatare moving in the vicinity of the user. As part of block 3055, the usermay be invited to retry the sleep tracking setup process. If the userretries, method 3000 may return to block 3010.

If block 3030 is evaluated in the affirmative, method 3000 may proceedto block 3035. At block 3035, block 3025 may continue to be performedsuch that a current classification is determined and stored. Over awindow of time, a determination may be made at block 3040 whether theclassifications stored at block 3035 indicate that the user is presentand static for at least a defined threshold amount of the window of time(or some other time-based threshold criterion is met that indicates theuser was sufficiently present and static). If not, method 3000 proceedsto block 3055. If block 3040 is determined in the affirmative, sleeptracking can be activated at block 3045. Sleep tracking and other healthmonitoring may then be automatically performed when the user isidentified as present in bed (assuming the user has properly consentedto such monitoring). Sleep tracking may be more likely to capture usefuldata since the user has conducted the sleep tracking setup process andensured that the user sleeps the correct distance from the device, thedevice is aimed correctly, and moving objects near where the user sleepshave been removed.

At block 3050, an indication may be output to the user indicating thatthe sleep tracking was successfully setup. This can include an auditorymessage (e.g., synthesized speech) being output indicating the successand/or a graphical user interface being presented that indicates thatsetup was successful.

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail in order to avoidobscuring the configurations. This description provides exampleconfigurations only, and does not limit the scope, applicability, orconfigurations of the claims. Rather, the preceding description of theconfigurations will provide those skilled in the art with an enablingdescription for implementing described techniques. Various changes maybe made in the function and arrangement of elements without departingfrom the spirit or scope of the disclosure.

Also, configurations may be described as a process which is depicted asa flow diagram or block diagram. Although each may describe theoperations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be rearranged. A process may have additional steps notincluded in the figure. Furthermore, examples of the methods may beimplemented by hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware, or microcode, the programcode or code segments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of the invention.Also, a number of steps may be undertaken before, during, or after theabove elements are considered.

What is claimed is:
 1. A contactless cough detection device, comprising:a housing; a microphone housed by the housing; a wireless networkinterface housed by the housing; an electronic display housed by thehousing, wherein the electronic display is visible through a screen; aradar sensor housed by the housing, wherein the radar sensor emits radiowaves through the screen; and a processing system housed by the housing,comprising one or more processors, that receives data from themicrophone and the radar sensor and is in communication with thewireless network interface and electronic display, wherein theprocessing system is configured to: receive audio data from themicrophone; detect that a cough has occurred based on the received audiodata; receive radar data indicative of reflected radio waves from theradar sensor; perform a state analysis process based on the receivedradar data for a user within a bed to select a state from a plurality ofstates; attribute the detected cough to the user within the bed based atleast in part on the state selected for the user based on the receivedradar data; cause cough data for the user corresponding to theattributed detected cough to be stored; and output the stored cough datavia the electronic display.
 2. The contactless cough detection device ofclaim 1, wherein the processing system detects that the cough hasoccurred by analyzing the received audio data from the microphone usinga trained cough detection machine learning model.
 3. The contactlesscough detection device of claim 1, wherein the processing system isfurther configured to delete the audio data received from the microphoneafter detecting that the cough has occurred.
 4. The contactless coughdetection device of claim 1, wherein the processing system beingconfigured to perform the state analysis process comprises theprocessing system being configured to determine the user has movedwithin the bed within a time period of the detected cough.
 5. Thecontactless cough detection device of claim 1, wherein the processingsystem is further configured to cause sleep data indicative of the coughbeing attributed to the user to be stored.
 6. The contactless coughdetection device of claim 5, wherein the processing system is furtherconfigured to transmit the sleep data to a remote server system via thewireless network interface.
 7. The contactless cough detection device ofclaim 1, wherein the processing system being configured to output thestored cough data via the electronic display comprises the processingsystem being configured to output a report indicating a number of timesthat the user coughed during sleep.
 8. The contactless cough detectiondevice of claim 1, wherein: the radar sensor is an integrated circuit(IC) distinct from the processing system; and the radar sensor outputsfrequency-modulated continuous wave (FMCW) radar into an environment ofthe contactless cough detection device.
 9. The contactless coughdetection device of claim 8, wherein the FMCW radar has a frequencybetween 57 and 64 GHz and has a peak effective isotropically radiatedpower (EIRP) of 20 dBm or less.
 10. The contactless cough detectiondevice of claim 1, wherein the radar sensor emits the radio wavesthrough an opaque bezel of the screen.
 11. The contactless coughdetection device of claim 1, wherein the processing system is furtherconfigured to determine that the detected cough woke the user from sleepbased on a change in the state of the user.
 12. The contactless coughdetection device of claim 1, wherein the processing system beingconfigured to output the stored cough data comprises the processingsystem being configured to output a report indicative of a number oftimes the user coughed during a night to the electronic display forpresentation.
 13. The contactless cough detection device of claim 1,wherein the processing system being configured to output the storedcough data comprises the processing system being configured to outputthe cough data with sleep data for the user to the electronic displayfor presentation.
 14. The contactless cough detection device of claim 1,further comprising a speaker housed by the housing.
 15. A method forperforming contactless cough detection, the method comprising:receiving, by a contactless cough detection device, an audio data streamfrom a microphone; detecting, by the contactless cough detection device,that a cough has occurred based on the received audio data stream;receiving, by the contactless cough detection device, a radar datastream indicative of reflected radio waves from a radar sensor;performing, by the contactless cough detection device, a state analysisprocess based on the received radar data such that a state is selectedfrom a plurality of states for a user within a bed; attributing, by thecontactless cough detection device, the detected cough to the userwithin the bed based at least in part on the state selected based on thereceived radar data; storing, by the contactless cough detection device,cough data indicating the attributed detected cough for the user; andoutputting, by the contactless cough detection device, the stored coughdata via an electronic display, wherein: the electronic display isvisible through a screen of the contactless cough detection device; andthe radar sensor emits and senses radar waves through the screen of thecontactless cough detection device.
 16. The method for performingcontactless cough detection of claim 15, wherein detecting that thecough has occurred is performed by analyzing the received audio datastream using a pre-trained cough detection machine learning model. 17.The method for performing contactless cough detection of claim 16, themethod further comprising: deleting the received audio data stream afterdetecting that the cough has occurred.
 18. The method for performingcontactless cough detection of claim 15, wherein performing the stateanalysis process comprises determining that the user has moved withinthe bed within a time period of the detected cough.
 19. The method forperforming contactless cough detection of claim 15, further comprising:determining that the detected cough woke the user from sleep based on achange in the state for the user.
 20. The method for performingcontactless cough detection of claim 15, wherein outputting the storedcough data comprises outputting the cough data with sleep data forpresentation.